# data[].attributes.attachment_link.readable_text
A few weeks ago, Bloomberg published a damning piece about how the DCI Group, a D.C.-based consulting firm, creates phony echo chambers to help push its clients' interests. For example, it created a 'coalition' of groups to support its campaign against Argentina on behalf of the country's Wall Street bondholders, who wanted their money back. (The coalition included some groups that didn't know they were in the coalition.)

This is a normal practice in Washington, as normal as going to Le Diplomate or think tank events on civility featuring war criminals. It's called astroturfing: the practice of creating the impression of widespread support for an issue on behalf of a narrow subset of interests, or even one individual, by creating fake or semi-fake organizations, petitions, rallies-whatever makes it seem like the activity isn't coming from its true source. The Tea Party movement was full of examples of this activity.

Astroturfing is easier now than ever before, and also easier than ever to conceal. It's very easy to set up a website, a Facebook page, or just a Twitter account, and pose as a Real Organization that does the things legitimate public policy organizations do: getting quoted in the news, releasing reports, generally doing actual activities to study public policy (even if those organizations are themselves often corrupt).

It also turns out to be even easier than I thought to do this on Twitter. Over the past year or so, Twitter users have noticed at least two bizarre ad campaigns that don't link to a real Twitter account, or have any presence on the web at all: Heartland Priorities and the Middle America Project. Using a Twitter Ads feature that allows advertisers to create promoted tweets that aren't linked to any permanent profile, these fake organizations are running ads on a number of important public policy issues. Thanks to Twitter's complete lack of transparency (and refusal to comment for this story), we have no idea who they are.

Heartland Priorities has run a few of these mystery ads, all about Scott Pruitt. A friend sent me these examples that appeared on his feed: (My texts with friends are normal and cool.)

The ads seem to have appeared around November last year.

There is a Heartland Priorities based in Missouri. Its founder told me that it has nothing to do with these ads; he works on marijuana and criminal justice reform. As far as I can tell, the Heartland Priorities of this Twitter ad simply doesn't exist. There is no website, nothing in corporate records (except the other Heartland Priorities), no nonprofit 990 forms on file with the IRS. So whoever runs this decided it would be better-more significant, more persuasive-to create a fake group with a convincing name, Heartland Priorities, instead of buying these ads as Jim The Guy Who Hates Scott Pruitt-or, more likely, not a guy, but a lobbying, consulting, or advocacy organization, or someone in their employ.

Stranger still are the Twitter ads run by the so-called Middle America Project. Like Heartland Priorities, the Middle America Project ads are just links to news articles, with a few words of commentary. But its ads take on a lot of different causes: pro-immigration, pro-NAFTA, anti-drug importation.

Many Twitter users have noted their ads on net neutrality.

One ad, for example, says broadband investment will go up without net neutrality in the way (which is, by the way, extremely misleading). It also ran an ad promoting an article about the lack of regulation and concentration of power in the tech industry, and another highlighting a piece about Google's lobbying spending. (In general, the tech industry largely supports net neutrality, while the cable industry has strongly opposes it.)

The organization, or whatever it is, also seems concerned with defending conservative Democrats. One ad complained about attacking vulnerable Democrats, decreeing it a "#wasteoftime," according to a tweet from April last year.

While another ad contained a link to a list of "things progressives should do before primarying Joe Manchin":

Its views on tax policy seem to be mixed. One ad posted a Vox article on how the Republican tax plan would raise taxes on families who adopt children, but another, in August 2017, tweeted a Bloomberg piece saying there's "a lot to like about cutting corporate taxes."

So, who is Middle America Project, and what do they want? I have absolutely no idea.

There are two Middle America Projects to be found in corporate records: Middle America Project LLC, founded on March 30, 2017, and Middle America Project, Inc, founded on May 12, 2017. Both were registered in Delaware by Corporation Service Company, which means those records tell us nothing about who actually runs the organization; this is perfectly legal in Delaware and a few other states.

Other than these promoted tweets, that is the only record of Middle America Project I can find, and it tells us nothing. They have no website, meaning no contact information, no list of staffers, no mission statement. There's no Twitter account associated with the tweets. The corporate records are anonymous. It could be anyone. It could be some oddball individual who's decided he wants to run Twitter ads as a fake organization called Middle America Project just to share interesting links that he likes. Or, it could be a (very lame, low-rent) astroturfing effort by an existing organization, one that loves NAFTA and opposes net neutrality and thinks everyone should stop being mean to conservative Democrats.

Astroturfing is a very useful technique for lobbyists, advocacy groups, Koch brothers types, corporations, and other slimy creatures of the night that seek to influence public policy. By creating the impression of broad popular support, the true aims and motivations of the organization or individuals behind the campaign are obscured. If you're a billionaire and you want some arcane tax repealed because it'll save you, personally, millions of dollars, that alone doesn't make for a particularly compelling public-facing argument. But if you can magic up a few thousand Real Americans who also appear to want the tax repealed, it doesn't just obscure your motives, it gives lawmakers cover to support something that primarily just benefits you, the rich person.

But these odd Twitter ads aren't even really astroturfing, in the traditional sense of the word. The campaign isn't designed to create the impression of mass grassroots or academic support for some issue. Not only is there no faked petition or list of supporting organizations, there isn't even a website, or an organization to speak of. It certainly isn't as significant as, for example, what DCI did with its Argentina-bashing campaign, which Bloomberg reported on. That was at least an echo chamber with real human voices, testifying to Congress and dressing up as a big rat. This is just bizarre Twitter ads linking to news stories. In the old days, you had to at least buy a website or get a PO box address to astroturf. Now, you just need a logo and a generic name, and you too can play at being a public policy organization.

And Twitter makes this very easy. It has a feature called Ads Without Profiles, which allows brands to advertise without actually having a Twitter account. It is sometimes used for, shall we say, "legitimate" advertising purposes-HBO used it to promote Silicon Valley, for example. A Twitter Ads pamphlet, posted on the website of the health marketing company Klick, boasts "Brand Safety" as a benefit of this feature, because users can't interact with the post-nasty Twitter users can't own you and your stupid ads, basically, thus protecting the Brand. The feature isn't available by default on Twitter's ad dashboard. Twitter has to specifically grant access.

So, someone has signed up for Twitter Ads posing as a completely made-up organization, or Twitter has decided that whoever is pretending to be the Middle America Project is allowed to pose as a completely made-up organization on their feed. Does that matter to them, at all? How hard would it be for a foreign government interested in influencing American democracy to purchase ads posing as a fake American public policy organization and promoting a policy goal, or attacking politicians? Much has been made of Twitter's fake Russian profiles and "bots," but is anything being done to prevent Russia, or any nefarious actor, from using these totally anonymous ads in misinformation or propaganda campaigns? For that matter, what prevents Coke from creating anonymous ads pretending to be concerned beverage scientists warning users that Pepsi contains unacceptably high levels of rat urine?

Unfortunately, we can't get any answers on these questions from Twitter. The company declined to comment for this story. Instead, they pointed us to their new rules for political ad disclosure, which promise "New Transparency For Ads On Twitter." Of course, these rules wouldn't apply to many of these mystery tweets, because they apply only to ads referring to "a clearly identified candidate (or party associated with that candidate) for any elected office."

And because neither Heartland Priorities nor Middle America Project actually has a Twitter account or a website, we are stuck with no way of knowing the extent of their campaign, what their policy goals actually are, or whether they're the product of one person or a huge organization. They exist only in the Twitter ad world, and Twitter won't tell us anything about them. The only evidence we have of their existence is tweeted screenshots of their ads, which surely only represent a small fraction of the actual number of ads served. Most people won't tweet about it when they get an ad like that. Many might not even notice that it's an ad.

In October, Twitter announced it would create a new "transparency center," promising, "in the coming weeks," to give users the ability to "see who had created an ad, how long an ad had been running and how the ad was targeted toward a specific user." It's now almost four months later, and still no transparency center. It's hard to see how any transparency promises square with Twitter's Ads Without Profiles option, which seems designed to evade transparency.

There are, of course, no laws that compel Twitter to release any information at all about people or organizations running ads on their service posing as fake organizations. That, in America, is perfectly fine. All we can do is ask if they know who these organizations are, and if they'd tell us if they did.

Don't hold your breath.

# data[].attributes.attachment_link.title
Twitter Has an Astroturfing Tool and They Won't Tell Us Who's Using It

# data[].attributes.attachment_link.description
A few weeks ago, Bloomberg published a damning piece about how the DCI Group, a D.C.-based consulting firm, creates phony echo chambers to help push its clients' interests. For example, it created a 'coalition' of groups to support its campaign against Argentina on behalf of the country's Wall Street bondholders, who wanted their money back.

# data[].attributes.attachment_link.title
The Future of the Internet and How to Stop It

# data[].attributes.headline
Problem 3: Unintended negative consequences

# data[].attributes.subject
Problem 3: Unintended negative consequences

# data[].attributes.headline
Problem 2: System design with perverse incentives

# data[].attributes.subject
Problem 2: System design with perverse incentives

# data[].attributes.attachment_link.title
Could this be the solution to stop the spread of fake news?

# data[].attributes.attachment_link.description
The best solution to stop the spread false political news stories and misinformation could be crowdsourcing.

# data[].attributes.attachment_link.readable_text
Today, 30 years on from my original proposal for an information management system , half the world is online. It's a moment to celebrate how far we've come, but also an opportunity to reflect on how far we have yet to go.

The web has become a public square, a library, a doctor's office, a shop, a school, a design studio, an office, a cinema, a bank, and so much more. Of course with every new feature, every new website, the divide between those who are online and those who are not increases, making it all the more imperative to make the web available for everyone.

And while the web has created opportunity, given marginalised groups a voice, and made our daily lives easier, it has also created opportunity for scammers, given a voice to those who spread hatred, and made all kinds of crime easier to commit.

Against the backdrop of news stories about how the web is misused, it's understandable that many people feel afraid and unsure if the web is really a force for good. But given how much the web has changed in the past 30 years, it would be defeatist and unimaginative to assume that the web as we know it can't be changed for the better in the next 30. If we give up on building a better web now, then the web will not have failed us. We will have failed the web.

To tackle any problem, we must clearly outline and understand it. I broadly see three sources of dysfunction affecting today's web:

Deliberate, malicious intent, such as state-sponsored hacking and attacks, criminal behaviour, and online harassment.

System design that creates perverse incentives where user value is sacrificed, such as ad-based revenue models that commercially reward clickbait and the viral spread of misinformation.

Unintended negative consequences of benevolent design, such as the outraged and polarised tone and quality of online discourse.

While the first category is impossible to eradicate completely, we can create both laws and code to minimize this behaviour, just as we have always done offline. The second category requires us to redesign systems in a way that change incentives. And the final category calls for research to understand existing systems and model possible new ones or tweak those we already have.

You can't just blame one government, one social network or the human spirit. Simplistic narratives risk exhausting our energy as we chase the symptoms of these problems instead of focusing on their root causes. To get this right, we will need to come together as a global web community.

At pivotal moments, generations before us have stepped up to work together for a better future. With the Universal Declaration of Human Rights, diverse groups of people have been able to agree on essential principles. With the Law of Sea and the Outer Space Treaty, we have preserved new frontiers for the common good. Now too, as the web reshapes our world, we have a responsibility to make sure it is recognised as a human right and built for the public good. This is why the Web Foundation is working with governments, companies and citizens to build a new Contract for the Web .

This contract was launched in Lisbon at Web Summit, bringing together a group of people who agree we need to establish clear norms, laws and standards that underpin the web. Those who support it endorse its starting principles and together are working out the specific commitments in each area. No one group should do this alone, and all input will be appreciated. Governments, companies and citizens are all contributing, and we aim to have a result later this year.

Governments must translate laws and regulations for the digital age. They must ensure markets remain competitive, innovative and open. And they have a responsibility to protect people's rights and freedoms online. We need open web champions within government - civil servants and elected officials who will take action when private sector interests threaten the public good and who will stand up to protect the open web.

Companies must do more to ensure their pursuit of short-term profit is not at the expense of human rights, democracy, scientific fact or public safety. Platforms and products must be designed with privacy, diversity and security in mind. This year, we've seen a number of tech employees stand up and demand better business practices. We need to encourage that spirit.

And most important of all, citizens must hold companies and governments accountable for the commitments they make, and demand that both respect the web as a global community with citizens at its heart. If we don't elect politicians who defend a free and open web, if we don't do our part to foster constructive healthy conversations online, if we continue to click consent without demanding our data rights be respected, we walk away from our responsibility to put these issues on the priority agenda of our governments.

The fight for the web is one of the most important causes of our time. Today, half of the world is online. It is more urgent than ever to ensure the other half are not left behind offline, and that everyone contributes to a web that drives equality, opportunity and creativity.

The Contract for the Web must not be a list of quick fixes but a process that signals a shift in how we understand our relationship with our online community. It must be clear enough to act as a guiding star for the way forward but flexible enough to adapt to the rapid pace of change in technology. It's our journey from digital adolescence to a more mature, responsible and inclusive future.

The web is for everyone and collectively we hold the power to change it. It won't be easy. But if we dream a little and work a lot, we can get the web we want.
Sir Tim There are many ways you can support the open web on its 30th birthday: En el cumpleaños número 30 de la World Wide Web (www.), nuestro fundador e inventor de la web Sir Tim Berners-Lee reflexiona sobre cómo la web ha transformado nuestro mundo y qué debemos hacer para construir una web que sea mejor y sirva a la humanidad. Los invitamos a compartir esta nota utilizando los hashtags #Web30 #ForTheWeb.
Hoy, 30 años después de aquel documento en el que proponía un sistema para la administración de información que hoy conocemos como , la mitad del mundo está en línea. Es momento de celebrar lo mucho que hemos avanzado, pero también representa una oportunidad para reflexionar sobre lo mucho que aún queda por hacer.

La web se ha convertido en una plaza pública, una biblioteca, un consultorio médico, una tienda, una escuela, un estudio de diseño, una oficina, un cine, un banco y mucho más. Por supuesto, con cada nueva función, cada nuevo sitio web, aumenta la brecha entre aquellos que están conectados y aquellos que no lo están, por lo que resulta imprescindible hacer que la web esté disponible para todo el mundo.

Y aunque la web ha creado oportunidades, ha dado voz a grupos marginados y ha hecho más fácil nuestra vida cotidiana, también ha creado oportunidades para estafadores, ha dado voz a aquellos que propagan el odio y ha facilitado la comisión de todo tipo de delitos.

En tiempos en que artículos sobre el uso indebido de la web dominan el ciclo noticioso, es comprensible que a muchas personas la web les genere temor, y no tengan la certeza de que realmente sea una fuerza positiva. Pero, en vista de lo mucho que ha cambiado la web en los últimos 30 años, sería derrotista y poco imaginativo suponer que la web tal como la conocemos no puede mejorar en los próximos 30 años. Si renunciamos a mejorarla ahora, no será la web la que nos habrá fallado. Seremos nosotros los que le habremos fallado a la web.

Para afrontar cualquier problema, debemos explicarlo y comprenderlo claramente. En términos generales, considero que la web actual se ve afectada por tres fuentes de disfuncionalidad:

Intentos maliciosos y deliberados, como la piratería, los ataques informáticos patrocinados por un estado, las conductas delictivas y el acoso en línea.

Sistemas que por diseño crean incentivos perversos y sacrifican los intereses del usuario, como los modelos de negocio basados en la publicidad que recompensan comercialmente el y la viralización de información falsa.

Diseños benevolentes que, sin embargo, y de manera involuntaria, generan consecuencias negativas, como el tono y la calidad atroz y polarizada del discurso en línea actual.

Si bien es imposible erradicar por completo los males de la primera categoría, podemos elaborar leyes y código para minimizar este comportamiento, tal como siempre lo hemos hecho fuera de línea. La segunda categoría nos exige rediseñar los sistemas de manera que cambien los incentivos. Y la última categoría requiere más investigación para comprender los sistemas existentes y modelar posibles nuevos sistemas o modificar los que ya tenemos.

No podemos limitarnos a culpar a un gobierno, una red social o al espíritu humano. Las narrativas simplistas generan el riesgo de que se agote nuestra energía mientras perseguimos los síntomas de estos problemas en lugar de concentrarnos en sus raíces. Para hacerlo bien, necesitamos unirnos como comunidad global.

En momentos cruciales, las generaciones que nos han precedido han dado un paso adelante para trabajar juntas por un futuro mejor. Con la Declaración Universal de los Derechos Humanos, diversos grupos de personas han sido capaces de ponerse de acuerdo sobre los principios básicos. Con el Tratado sobre el espacio exterior y la Convención sobre el Derecho del Mar, hemos conservado nuevas fronteras para el bien común. Ahora, a medida que la web transforma nuestro mundo, también tenemos la responsabilidad de asegurarnos de que se reconozca como un derecho humano y se desarrolle para el bien común. Por esta razón, la Web Foundation está trabajando con los gobiernos, empresas y ciudadanos para elaborar un nuevo contrato para la web.

Este Contrato se lanzó en la Web Summit que se llevó a cabo en Lisboa, donde se reunió un grupo de personas que están de acuerdo en que se necesita establecer normas, leyes y estándares claros que sustenten la web. Aquellos que ya se suscribieron avalan sus principios básicos y están desarrollando juntos los compromisos específicos de cada área. Ningún grupo debería hacer avanzar de manera unilateral, por lo que se agradece cualquier contribución. Contamos con la colaboración de gobiernos, empresas y ciudadanos, y esperamos tener el resultado a finales de este año.

Los gobiernos deben adaptar las leyes y reglamentos para la era digital. Deben garantizar que los mercados sigan siendo competitivos, innovadores y abiertos. Además, tienen la responsabilidad de proteger los derechos y las libertades de las personas en línea. Necesitamos contar con defensores de la web abierta en los gobiernos, empleados públicos y funcionarios electos que intervengan cuando los intereses del sector privado amenacen el bien común, y que se alcen para proteger la web abierta.

Las empresas deben hacer más para garantizar que su búsqueda de beneficios a corto plazo no sea a costa de los derechos humanos, la democracia, los hechos científicos o la seguridad pública. Tanto las plataformas como los productos deben ser diseñados teniendo en cuenta la privacidad, la diversidad y la seguridad. Este año, hemos visto que varios empleados de empresas tecnológicas se han alzado para exigir mejores prácticas comerciales. Necesitamos alentar ese espíritu.

Y lo más importante de todo, los ciudadanos deben hacer rendir cuentas a las empresas y a los gobiernos por los compromisos asumidos, y exigir que respeten la web como una comunidad global que tiene a los ciudadanos por centro. Si no elegimos políticos que defiendan a la web como un espacio libre y abierto, si no hacemos nuestro aporte para alentar conversaciones constructivas y saludables en línea, si seguimos haciendo click en sin exigir que se respeten nuestros derechos sobre nuestros datos, eludimos nuestra responsabilidad de poner estos temas en entre las prioridades de nuestros gobiernos.

La lucha por la web es una de las causas más importantes de nuestra era. Actualmente, la mitad del mundo está en línea. Es más urgente que nunca asegurarnos de que la otra mitad no se quede afuera, y de que todos contribuyan para tener una web que impulse la igualdad, las oportunidades y la creatividad.

El Contrato para la Web no debe ser una lista de soluciones rápidas, sino un proceso que señale un cambio en cómo concebimos nuestra relación con nuestra comunidad en línea. Debe ser lo suficientemente claro para actuar como una guía para el futuro, pero también lo suficientemente flexible para adaptarse al ritmo acelerado de los cambios en la tecnología. Representa nuestro recorrido desde la adolescencia digital a un futuro más maduro, responsable e inclusivo.

La web es para todos y colectivamente tenemos el poder para cambiarla. No será fácil. Pero si soñamos un poco y trabajamos mucho, podemos obtener la web que deseamos.
Sir Tim Existen muchas maneras en que puedes apoyar la web abierta en su 30 aniversario: A l'occasion du trentenaire du web, notre Fondateur et l'Inventeur du web, Sir Tim Berners-Lee réfléchit sur le changement apporté à notre monde par le web et ce qu'il faut faire pour construire un meilleur web qui est au service de l'humanité. Merci de partager en utilisant #Web30 et #ForTheWeb
Aujourd'hui, trente ans après ma proposition initiale de système de gestion de l'information, la moitié du monde est en ligne. C'est le moment de célébrer le chemin parcouru, mais aussi de réfléchir au chemin qu'il nous reste à parcourir.

Le Web est devenu une place publique, une bibliothèque, un cabinet de médecin, un magasin, une école, un studio de design, un bureau, un cinéma, une banque et bien plus encore. Bien sûr, avec chaque nouvelle fonctionnalité, chaque nouveau site web, le fossé entre ceux qui sont en ligne et ceux qui ne le sont pas augmente. Il est donc d'autant plus impératif de rendre le Web accessible à tous.

Et si le Web a créé des opportunités, donné une voix aux groupes marginalisés et simplifié notre vie quotidienne, il a également créé des opportunités pour les fraudeurs, et donné une voix à ceux qui propagent la haine et facilitent la perpétration de toutes sortes de crimes.

Sur fond de reportages sur la mauvaise utilisation du Web, il est compréhensible que de nombreuses personnes aient peur et se demandent si le Web est vraiment une force au service du bien. Mais compte tenu de tout ce que le Web a changé au cours des trente dernières années, il serait défaitiste et dénué d'imagination de supposer que le Web tel que nous le connaissons ne peut pas être amélioré dans les trente prochaines années. Si nous renonçons à construire un Web meilleur aujourd'hui, ce n'est pas le Web qui nous aura fait défaut. C'est nous qui aurons fait défaut au Web.

Pour résoudre tout problème, il faut le décrire et le comprendre clairement. Je vois en gros trois sources de dysfonctionnement qui affectent le Web d'aujourd'hui :

Les intentions délibérées et malveillantes, comme le piratage et les attaques d'État, les comportements criminels et le harcèlement en ligne.

Une conception de système qui crée des incitations perverses dans lesquelles la valeur des utilisateurs est sacrifiée, comme des modèles de revenus basés sur la publicité qui récompensent commercialement les pièges à clics et la propagation virale de la désinformation.

Les conséquences négatives involontaires d'une conception bienveillante, telles que le ton indigné et divergent, et la qualité du discours en ligne.

La première catégorie est impossible à éradiquer complètement, mais il est possible de créer un code et des lois pour minimiser ces comportements, comme cela a toujours été le cas hors ligne, dans le monde réel. La deuxième catégorie nous oblige à repenser les systèmes de manière à modifier les incitations. Et la dernière catégorie nécessite des recherches pour comprendre les systèmes existants et modéliser de nouveaux systèmes possibles, ou modifier ceux dont nous disposons déjà.

On ne peut pas se contenter de tenir pour responsable un gouvernement, un réseau social ou l'esprit humain. Les récits simplistes risquent d'épuiser notre énergie tandis que nous poursuivons les symptômes de ces problèmes au lieu de nous concentrer sur leurs causes profondes. Pour agir comme il faut, nous devrons nous rassembler en tant que communauté web mondiale.

À certains moments cruciaux, les générations qui nous ont précédés se sont mobilisées pour travailler en commun à un avenir meilleur. Avec la Déclaration universelle des droits de l'homme, divers groupes de personnes ont pu se mettre d'accord sur des principes essentiels. Avec le droit de la mer et le traité de l'espace, nous avons préservé de nouvelles frontières pour le bien commun. Aujourd'hui également, à mesure que le Web remodèle notre monde, nous avons la responsabilité de nous assurer que celui-ci soit reconnu comme un droit fondamental et construit pour le bien public. C'est pourquoi la Fondation Web collabore avec les gouvernements, les entreprises et les citoyens pour créer un nouveau Contrat pour le Web.

Ce contrat a été lancé à Lisbonne lors du Web Summit, réunissant un groupe de personnes qui sont d'avis que nous devons établir des normes, lois et standards clairs qui sous-tendent le Web. Ceux qui le soutiennent souscrivent à ses principes de départ et élaborent ensemble des engagements spécifiques dans chaque domaine. Aucun groupe ne doit y travailler seul, et toute contribution sera appréciée. Les gouvernements, les entreprises et les citoyens contribuent tous, et notre but est d'arriver à un résultat dans le courant de cette année.

Les gouvernements doivent traduire les lois et les réglementations à l'ère numérique. Ils doivent s'assurer que les marchés restent compétitifs, innovants et ouverts. Ils ont la responsabilité de protéger les droits et les libertés des personnes en ligne. Nous avons besoin de champions du Web ouvert au sein des gouvernements - de fonctionnaires et d'élus qui prendront des mesures lorsque les intérêts du secteur privé menaceront le bien public et qui défendront le Web ouvert.

Les entreprises doivent faire davantage pour s'assurer que la recherche du profit à court terme ne se fasse pas au détriment des droits humains, de la démocratie, des faits scientifiques ou de la sécurité publique. Les plates-formes et les produits doivent être conçus dans le respect de la confidentialité, de la diversité et de la sécurité. Cette année, plusieurs employés du secteur de la technologie ont pris position pour réclamer de meilleures pratiques commerciales. Nous devons encourager cet esprit.

Et surtout, les citoyens doivent tenir les entreprises et les gouvernements pour responsables de leurs engagements et exiger qu'ils respectent le Web en tant que communauté mondiale dont les citoyens sont au cœur. Si nous n'élisons pas des politiciens qui défendent un Web libre et ouvert, si nous ne faisons pas notre part pour favoriser des conversations en ligne constructives et saines, si nous continuons à donner notre consentement sans exiger que nos droits en matière de données soient respectés, nous nous déchargeons de notre responsabilité à inscrire ces questions à l'agenda des priorités de nos gouvernements.

La lutte pour le Web est l'une des causes les plus importantes de notre époque. Aujourd'hui, la moitié du monde est en ligne. Il est plus urgent que jamais de veiller à ce que l'autre moitié ne soit pas laissée de côté et à ce que tout le monde contribue à un Web propice à l'égalité, aux opportunités et à la créativité.

Le contrat pour le Web ne doit pas être une liste de solutions rapides, mais un processus qui marque un changement dans la manière dont nous comprenons notre relation avec notre communauté en ligne. Il doit être suffisamment clair pour servir de guide pour la voie à suivre, mais suffisamment souple pour s'adapter au rythme rapide des changements technologiques. C'est notre passage de l'adolescence numérique à un avenir plus mature, responsable et inclusif.

Le Web est à tous, et nous détenons collectivement le pouvoir de le changer. Ce ne sera pas facile. Mais en rêvant un peu et en travaillant beaucoup, nous pouvons créer le Web que nous voulons.
Sir Tim Il y a plusieurs manières de démontrer votre soutien pour le web ouvert pour ce 30ème anniversaire: No 30º Anivers rio da World Wide Web. o nosso fundador e inventor da web Sir Tim Berners-Lee reflete sobre como a web transformou o nosso mundo e o que temos que fazer para construir uma web melhor para servir a humanidade. Por favor partilhe esta nota usando #Web30 #PelaWeb
Hoje, 30 anos depois de minha proposta original para um sistema de gerenciamento de informações, metade do mundo está on-line. É o momento para celebrar o quão longe chegamos, mas também uma oportunidade para refletir sobre até onde temos de ir ainda.

A web se transformou em praça pública, biblioteca, consultório médico, loja, escola, estúdio de design, escritório, cinema, banco e muito mais. É claro que com cada novo recurso, cada novo site, a divisão entre os que estão on-line e os que não estão, vai aumentando, tornando ainda mais imperativo fazer da web um local disponível para todo o mundo.

E ao mesmo tempo que a web foi criando oportunidades, dando voz aos grupos marginalizados e facilitando nossas vidas diárias, também criou oportunidades para golpistas, dando voz àqueles que espalham o ódio e tornando mais fácil a perpetração de todos os tipos de crime.

Tendo em conta o pano de fundo das notícias que relatam como a web é mal utilizada, é compreensível que muitas pessoas sintam medo e insegurança, e se questionem se a web é realmente uma força do bem. Mas vendo o quanto a web mudou nos últimos 30 anos, seria derrotista e pouco imaginativo presumir que a web como a conhecemos não pode ser modificada para melhor nos próximos 30 anos. Se desistirmos agora de construir uma web melhor, então a web não terá falhado conosco. Nós teremos falhado para com a web.

Intenções maliciosas e deliberadas, como invasões e ataques patrocinados pelo Estado, comportamento criminoso e assédio on-line.

Design de sistemas que criam incentivos perversos em que o valor do usuário é sacrificado, como modelos de receita baseados em anúncios que recompensam comercialmente o isco para o clique e a disseminação viral da desinformação.

Consequências negativas não intencionais do design benevolente, como o tom ultrajado e polarizado e a qualidade do discurso on-line.

Para resolver qualquer problema, devemos começar por delineá-lo e compreendê-lo claramente. De um modo lato, eu posso ver três fontes de disfunção que afetam a web de hoje:

Embora a primeira categoria seja impossível de erradicar completamente, podemos criar leis e códigos para minimizar esse comportamento, tal como sempre fizemos off-line. A segunda categoria nos obriga a redesenhar os sistemas, de forma a mudar os incentivos. E a categoria final exige pesquisas para entender os sistemas existentes e modelar novos possíveis, ou ajustar os que já temos.

Você não pode culpar apenas um governo, uma rede social ou o espírito humano. Narrativas simplistas correm o risco de esgotar nossa energia ao perseguirmos os sintomas desses problemas, em vez de nos concentrarmos em suas causas. Para corrigir isso, precisamos nos unir como uma comunidade global da web.

Em momentos cruciais, gerações antes de nós se juntaram para trabalhar juntas para um futuro melhor. Com a Declaração Universal dos Direitos Humanos, diversos grupos de pessoas puderam entrar em acordo quanto a princípios essenciais. Com a Lei do Mar e o Tratado do Espaço Exterior, preservamos novas fronteiras para o bem comum. Também agora, à medida que a web vai reformulando nosso mundo, temos a responsabilidade de garantir que ela seja reconhecida como um direito humano e construída para o bem público. É por isso que a Web Foundation está trabalhando com governos, empresas e cidadãos para construir um novo Contrato para a Web .

Esse contrato foi lançado em Lisboa, na Web Summit, reunindo um grupo de pessoas que concordam que precisamos estabelecer normas, leis e padrões claros que sustentem a web. Aqueles que o apoiam, se reveem em seus princípios iniciais e, juntos, estão elaborando os compromissos específicos em cada área. Não deve ser um só grupo a fazer isso sozinho e todos os comentários serão bem-vindos. Governos, empresas e cidadãos estão contribuindo, e nosso objetivo é ter um resultado ainda este ano.

Os governos devem traduzir leis e regulamentos para a era digital. Eles devem garantir que os mercados permanecem competitivos, inovadores e abertos. E eles têm a responsabilidade de proteger os direitos e liberdades das pessoas on-line. Precisamos de defensores de uma web aberta dentro do governo - funcionários públicos e autoridades eleitas que agirão quando os interesses do setor privado ameaçarem o bem público e se levantarão para proteger a rede aberta.

As empresas devem fazer mais para garantir que sua busca por lucros a curto prazo não aconteça à custa dos direitos humanos, da democracia, dos fatos científicos ou da segurança pública. Plataformas e produtos devem ser projetados tendo em mente a privacidade, diversidade e segurança. Nesse ano, vimos vários funcionários do mundo da tecnologia levantarem suas vozes e exigirem melhores práticas de negócios. Precisamos encorajar esse espírito.

E o mais importante de tudo é que os cidadãos responsabilizem as empresas e os governos pelos compromissos que assumem e exijam que ambos respeitem a web como uma comunidade global, cujo núcleo assenta nos cidadãos. Se nós não elegermos políticos que defendam uma web livre e aberta, se não fizermos nossa parte para promover conversas construtivas e saudáveis on-line, se continuarmos clicando em consentimentos sem exigir que nossos direitos sobre os dados sejam respeitados, nos afastaremos de nossa responsabilidade de colocar essas questões na agenda prioritária de nossos governos.

A luta pela web é uma das causas mais importantes do nosso tempo. Hoje, metade do mundo está on-line. É mais urgente do que nunca garantir que a outra metade não seja deixada para trás, off-line, e que todos contribuam para uma web que impulsione a igualdade, a oportunidade e a criatividade.

O Contrato para a Web não deve ser uma lista de soluções rápidas, mas um processo que sinalize uma mudança na forma como entendemos nosso relacionamento com nossa comunidade on-line. Deve ser claro o suficiente para atuar como uma estrela-guia para o caminho a seguir, mas flexível o suficiente para se adaptar ao ritmo acelerado de mudança na tecnologia. É a nossa jornada da adolescência digital para um futuro mais maduro, responsável e inclusivo.
Sir Tim
A web é para todos e, coletivamente, temos o poder de mudá-la. Não será fácil. Mas se sonharmos um pouco e trabalharmos muito, podemos conseguir a web que queremos.
[This Japanese version was not translated by the Web Foundation. With thanks to Takashi Omamyuda and Michio Toyohara who volunteered the translation.]
Há muitas formas como pode apoiar a web aberta no seu 30. Aniversá rio :

私が独自の 情報管理システムの提案 をして30年が経った今日、世界の半分はオンラインで結ばれています。私たちがどれほどの道のりを歩んできたかを祝うとともに、まだ私たちが成し遂げていない未来について考える良い機会でもあります。

ウェブは公共空間になったと同時に、図書館や診療所、店舗、学校、デザインスタジオ、オフィス、映画館、銀行といった様々なものになってきました。当然ながら新しい機能やウェブサイトが登場するたびにオンラインの人とそうでない人との隔たりが広がってしまい、誰もがウェブを使えるようにすることはますます必須になっています。

ウェブは様々な機会を創り出し、片隅に追いやられている人々も声を上げられるようにし、私たちの日常生活を助けてきた一方で、お金をだまし取ったり憎悪を広げようとしたりする人たちや様々な犯罪の温床にもなってきました。

ウェブの誤った使い方を伝える数々のニュースを背景に、多くの人々がウェブを恐れたり、世の中のためになるものか信頼できないと感じたりすることも理解できます。しかし、この30年間にウェブがいかに発展してきたかを考えると、私たちがこれからの30年間に、いまウェブとして知っているものを良い方向に変えられないと考えるのは敗北であり、創造力の欠如を意味します。もし今より良いウェブを作るのをあきらめたとしたら、それはウェブの失敗ではなく、私たちがウェブの信頼を裏切ったことになります。

どのような問題を解決するにしても、問題の概要を明確に理解する必要があります。私には今日のウェブに機能不全をもたらしている原因が大きく分けて3つあると見ています。

1)　国家ぐるみのハッキングや攻撃、犯罪行為やオンラインハラスメントのような計画的な悪意

2) 　ユーザーの利益を犠牲にして誤ったインセンティブを生み出すシステムデザイン、例えば広告売上モデルで商業的報酬を狙ったクリックベイトや偽情報の拡散

3)　暴力や偏見に満ちた調子や性質を持つオンラインでのやりとりが、優れたデザインに反して負の結果をもたらすこと

最初のカテゴリーを根絶するのは不可能ですが、私たちがオフラインでやってきたように、このような行為を最小限に抑える法律や規則を作ることはできます。2番目についてはシステムを再設計して、インセンティブを変える方向に持っていくことが必要です。そして最後のカテゴリーは現在のシステムについて理解を深める研究を呼びかけて新たなモデルを作るか、既存の仕組みに手を加える必要があります。

一つの政府や一つのソーシャルネットワーク、もしくは人間の心のせいにすれば良いというものではありません。ものごとの単純化は問題の根本原因から目をそらし、起きた現象ばかり追いかけさせて私たちの労力を浪費させるリスクをはらみます。問題に対して正しく向き合うには、私たちがグローバルなウェブコミュニティーとして一丸となる必要があります。

歴史の転換点において、人々は何世代も前からともにより良き未来のために力を合わせてきました。世界人権宣言によって、多様な人々のグループが重要な基本原則について合意してきました。国際海洋法や宇宙条約によって、私たちは公益のために新たなフロンティアを保護してきました。同様にウェブが世の中を変えつつある現在、私たちはそれが人権の一つであり、公共の利益のために作られるという認識を確かなものにする責務があります。これは「 ワールド・ワイド・ウェブ・ファウンデーション 」が各国の政府や企業、市民とともに「 ウェブのための協定（Contract for the Web） 」 を作ろうとしている理由です。

この協定はリスボンで開催されたウェブサミットで、ウェブを支える明確な行動規範、法律や規準の制定が必要と考える人々のグループが合意して発表しました。この趣旨に賛同する人々は、その理念を支持し、それぞれの分野で具体的な課題に取り組んでいます。これは一つのグループが単独で取り組む問題ではありません。どんな提案も大いに歓迎します。政府や企業、市民がともに貢献しており、今年後半には結果を出すことを目標にしています。

政府は法律や規制をデジタル時代に即して変えなければなりません。また政府は市場の競争やイノベーション、オープンさを保つ必要があります。さらに人々のオンライン上の権利や自由を守る責任もあります。私たちは政府内に「オープン・ウェブ・チャンピオン（open web champions）」と呼ばれる公職者や公選の役職者といった、民間企業の利害によって公益が脅されそうなときに開かれたウェブを守るため行動する人々を必要としています。

企業は人権や民主主義、科学的事実や公共の安全などを犠牲にしてまで短期的利益を追い求めないようにする必要があります。プラットフォームや製品はプライバシーや多様性、セキュリティを念頭に設計される必要があります。今年に入っていくつかの企業の技術者たちが商慣行の改善を求めて立ち上がりました。その心意気を応援しなければなりません。

そして最も大切なことは、市民が企業や政府の約束した取り組みや説明責任を果たせるようにすること、企業や政府がともに市民中心のグローバルコミュニティーとしてのウェブを尊重するよう求めることです。もし私たちが自由でオープンなウェブを守る政治家を選なかったり、オンラインでの建設的で健全なやりとりを育む環境を作らなかったり、私たちのデータに対する権利の尊重を求めずに同意のクリックをし続けたりすると、私たちは政府にこれらの問題を重要課題として取り上げさせるという責務から逃げてしまうことになります。

ウェブのための闘いは、いまの時代に最も重要な大義の一つです。今や世界の半分はオンラインで結ばれています。残りの半分がオフラインのまま置き去りにされることがないように、平等や機会、そして創造性をもたらすウェブにすべての人が貢献することはかつてないほどに緊急性を帯びています。

「ウェブのための協定」はその場しのぎの対策ではありません。私たちがオンラインコミュニティーとの関係をどう理解するかという変化を示すプロセスです。これは私たちを前へと導く星のように明確なものである一方、進化のペースが非常に速い技術に対応できるように柔軟である必要があります。これはデジタル時代の思春期からの成長を目指す、責任のある、あらゆる人々の未来への旅路と言えます。

ウェブはすべての人のためにあり、私たちは力を合わせて変えることができます。決して容易なことではありませんが、少しの夢を持って力を尽くせば、私たちが望むウェブにできるのです。

# data[].attributes.attachment_link.description
On the World Wide Web's 30th birthday, our founder and web inventor Sir Tim Berners-Lee reflects on how the web has changed our world and what we must do to build a better web that serves all of humanity. Please share using #Web30 #ForTheWeb. For updates about our work, sign

# data[].attributes.attachment_link.readable_text
I finally went on spring break as in "I actually got away from work and stuff". We took our first cruise. For a social and institutional economist with a critical bent that just loves to observe people and capitalism in the wild, let's just say that a cruise offers a target rich environment. I'll have more on that in some other post yet to be written. But what I did want to comment on was the news last week in the edtech sector.

Last week Advance acquired Turnitin, the notorious stealer of student intellectual property doing business under the guise of offering "plagiarism checking" services. Turnitin also has some other related businesses such an auto-grading service, etc. The price was apparently $1.75 billion dollars. That's billion with a b.

I'm not here to talk about how awful the pedagogy of mistrusting all students is or how it's immoral and unethical to steal/coerce student's copyrights away from them. These are all horrid aspects of Turnitin and among the reasons why I've always, for over a decade, opposed and fought against use of it at my school. I'm not here to talk about those aspects because people much more knowledgeable than I have been saying that a lot - people like Jesse Stommel and Sean Michael Morris. And the sale announcement last week has brought a lot of faculty and teacher anger out in public about the sale.

What I want to talk about is the numbers on this deal (I am an economist, after all). $1.75 billion for this company. That's a pretty hefty valuation. Especially for a company that really isn't that big and hasn't been a huge growth tear. Yes, it's been growing and the core product/service probably has a lot of room to expand internationally. But from what I can tell on the Interwebs, Turnitin probably has annual revenue in the $127.7 million range. That's million with a m. 1.75 B to buy an annual 127.7 M. That means Advance is paying approximately 14 times annual revenue to buy it. Valuation of a company as a multiple of the annual revenue is common way in finance of comparing whether a deal is highly valued or cheaply valued, especially for technology companies and startups. Turnitin isn't exactly a startup - it was founded circa 1998, but it's still a "tech" company. For a tech company 14 x revenue isn't out of range, but it's not cheap either. Tech companies, especially ones that are expected to grow fast don't usually have strong current earnings (profits), so revenues times a multiplier is used to estimate value. So what this valuation tells us is that Advance expects Turnitin to produce some very significant growth - probably much faster than Turnitin has achieved so far in it's 20 year life.

And that's what scares me. Advance isn't really just a "family-owned company" as some reports have it. Yes, the ownership of Advance is private and dominated by descendants of Newhouse family (think newspaper publishers). But "family-owned" sounds warm and fuzzy like the diner down the street where you get breakfast. Advance is a serious technology, publishing, and communications conglomerate. And they've got ambitions. And they're serious with their money. They think like venture investors. If they invest $1.75 billion, they'll expect to turn it into $100 billion or more. That's the game.

Even if they had modest ambitions and only wanted to turn Turnitin into (read those last three words again just for fun) a modest 10 or 20 billion dollar company, they have to do something big and different. Turnitin isn't getting to that range on it's own by doing what they currently do: call students cheats and check for plagiarism.

So how does Advance expect to get it's money back multiple times? I don't know. They didn't share their thoughts with me. But they did share them The Chronicle:

Chris Caren, chief executive of Turnitin, said the company's next step is to become a platform for colleges and high schools to submit all types of student assignments, digital or on paper. It would then use AI to help instructors review that work to, among other things, spot at-risk students and devise remediation plans. The company is also developing Turnitin's software to branch out into the STEM fields and detect plagiarism in coding, for example. In other words, it hopes to become a one-stop shop for all sorts of tech-driven teaching services.

...

Advance, which owns companies like Condé Nast, has recently begun investing in data and analytics companies, said Janine Shelffo, Advance's chief strategy and development officer. Turnitin's strong market presence and its advanced technologies, said Shelffo, make Turnitin a valuable investment. "There's a whole road map where we can see where tech innovation will increasingly power personalized learning and enhance outcomes for all students."

It's time we connect the dots, folks. Advance isn't just a newspaper publisher. They're adept at cookies, tracking of readers on the web, and data collection just as Facebook and Google are. They publish online magazines (Conde Nast, hello?). They publish Reddit and Arstechnica and other sites. They also have very capable big data analysis capabilities ( 1010data) that "transforms Big Data into smart insights to create the High-Definition Enterprise that can anticipate and respond to change" for 850 large companies.

In the Chronicle article, Sean Michael Morris rightly observes that Advance/Turnitin could develop profiles of students using their data and monetize that via marketing and advertising. That's true but I think we're missing the mark. We're being distracted by the fact the big two of early surveillance capitalism, Facebook and Google, have monetized their vast troves of surveillance data by using it to sell advertising. Rule of thumb in business strategy: there's only room for 2-3 big monsters in any particular big industry. I don't think Advance/Turnitin will go that way.

I think it will be worse. They're already pushing automated-grading systems and student "feedback" systems. There's no technological difference between a system that checks a student's written posting or submission for plagiarism against a database of collected writings and a system that checks those same posts/submissions against a database of "approved" thoughts and phrases. In a way, isn't that what grading is anyway? Advance/Turnitin can easily morph into the thought police.

But can thought policing be monetized? You bet. First, any nation that thinks a social credit system for controlling ordinary behaviors like jay-walking is a good idea, will love the idea of policing thoughts and utterances. That will be worth a pretty penny. Let's suppose that social credit would never take off in the US or Western Europe (an assumption I am loath to make). Many, many institutions of higher education will jump at such a system. It's just an extension of grading - until the database of vetoed utterances, ideas, or word strings is expanded to include controversial ideas. Are you going to tell me there aren't some religious based schools that would buy a system that automatically rejects papers or forum postings that suggest abortion is acceptable? Of course, the system won't just "reject", it will provide "feedback". Those controversial ideas can be handled automatically and merged with the "inclusive" repository of acceptable learning materials (free!) which is also, ironically, the exclusive source of learning materials.

For profit colleges will love the systems since they' ll allow further cutbacks on faculty. Replacing labor with capital investment is one the oldest tricks in the capitalist toolchest. And, those colleges will get automated stats and data "proving" their students learned! Not only were the students' papers "correct" but everything they've said on Reddit and other social forums since has conformed to the acceptable.

Forget social credit. We've got the possible (probable?) platform for thought control scoring.

By all means let's complain, scream, and object to the abuses of the Turnitin plagiarism model. But let's keep our eyes open for the next big data thing. That's no plagiarism checker. That's the teaching death star.

# data[].attributes.attachment_link.description
I finally went on spring break as in "I actually got away from work and stuff". We took our first cruise. For a social and institutional economist with a critical bent that just loves to observe people and capitalism in the wild, let's just say that a cruise offers a target rich environment.

# data[].attributes.attachment_link.readable_text
Facial recognition can log you into your iPhone, track criminals through crowds and identify loyal customers in stores.

The technology - which is imperfect but improving rapidly - is based on algorithms that learn how to recognize human faces and the hundreds of ways in which each one is unique.

To do this well, the algorithms must be fed hundreds of thousands of images of a diverse array of faces. Increasingly, those photos are coming from the internet, where they're swept up by the millions without the knowledge of the people who posted them, categorized by age, gender, skin tone and dozens of other metrics, and shared with researchers at universities and companies.

As the algorithms get more advanced - meaning they are better able to identify women and people of color, a task they have historically struggled with - legal experts and civil rights advocates are sounding the alarm on researchers' use of photos of ordinary people. These people's faces are being used without their consent, in order to power technology that could eventually be used to surveil them.

That's a particular concern for minorities who could be profiled and targeted, the experts and advocates say.

"This is the dirty little secret of AI training sets. Researchers often just grab whatever images are available in the wild," said NYU School of Law professor Jason Schultz.

The latest company to enter this territory was IBM, which in January released a collection of nearly a million photos that were taken from the photo hosting site Flickr and coded to describe the subjects' appearance. IBM promoted the collection to researchers as a progressive step toward reducing bias in facial recognition.

But some of the photographers whose images were included in IBM's dataset were surprised and disconcerted when NBC News told them that their photographs had been annotated with details including facial geometry and skin tone and may be used to develop facial recognition algorithms. (NBC News obtained IBM's dataset from a source after the company declined to share it, saying it could be used only by academic or corporate research groups.)

"None of the people I photographed had any idea their images were being used in this way," said Greg Peverill-Conti, a Boston-based public relations executive who has more than 700 photos in IBM's collection, known as a "training dataset."

"It seems a little sketchy that IBM can use these pictures without saying anything to anybody," he said.

John Smith, who oversees AI research at IBM, said that the company was committed to "protecting the privacy of individuals" and "will work with anyone who requests a URL to be removed from the dataset."

Despite IBM's assurances that Flickr users can opt out of the database, NBC News discovered that it's almost impossible to get photos removed. IBM requires photographers to email links to photos they want removed, but the company has not publicly shared the list of Flickr users and photos included in the dataset, so there is no easy way of finding out whose photos are included. IBM did not respond to questions about this process.

To see if your Flickr photos are part of the dataset, enter your username in a tool NBC News created based on the IBM dataset:

IBM says that its dataset is designed to help academic researchers make facial recognition technology fairer. The company is not alone in using publicly available photos on the internet in this way. Dozens of other research organizations have collected photos for training facial recognition systems, and many of the larger, more recent collections have been scraped from the web.

Some experts and activists argue that this is not just an infringement on the privacy of the millions of people whose images have been swept up - it also raises broader concerns about the improvement of facial recognition technology, and the fear that it will be used by law enforcement agencies to disproportionately target minorities.

"People gave their consent to sharing their photos in a different internet ecosystem," said Meredith Whittaker, co-director of the AI Now Institute, which studies the social implications of artificial intelligence. "Now they are being unwillingly or unknowingly cast in the training of systems that could potentially be used in oppressive ways against their communities."

How facial recognition has evolved

In the early days of building facial recognition tools, researchers paid people to come to their labs, sign consent forms and have their photo taken in different poses and lighting conditions. Because this was expensive and time consuming, early datasets were limited to a few hundred subjects.

With the rise of the web during the 2000s, researchers suddenly had access to millions of photos of people.

"They would go into a search engine, type in the name of a famous person and download all of the images," said P. Jonathon Phillips, who collects datasets for measuring the performance of face recognition algorithms for the National Institute of Standards and Technology. "At the start these tended to be famous people, celebrities, actors and sports people."

As social media and user-generated content took over, photos of regular people were increasingly available. Researchers treated this as a free-for-all, scraping faces from YouTube videos, Facebook, Google Images, Wikipedia and mugshot databases.

Academics often appeal to the noncommercial nature of their work to bypass questions of copyright. Flickr became an appealing resource for facial recognition researchers because many users published their images under "Creative Commons" licenses, which means that others can reuse their pictures without paying license fees. Some of these licenses allow commercial use.

To build its Diversity in Faces dataset, IBM says it drew upon a collection of 100 million images published with Creative Commons licenses that Flickr's owner, Yahoo, released as a batch for researchers to download in 2014. IBM narrowed that dataset down to about 1 million photos of faces that have each been annotated, using automated coding and human estimates, with almost 200 values for details such as measurements of facial features, pose, skin tone and estimated age and gender, according to the dataset obtained by NBC News.

It's a single case study in a sea of datasets taken from the web. According to Google Scholar, hundreds of academic papers have been written on the back of these huge collections of photos - which have names like MegaFace, CelebFaces and Faces in the Wild - contributing to major leaps in the accuracy of facial recognition and analysis tools. It was difficult to find academics who would speak on the record about the origins of their training datasets; many have advanced their research using collections of images scraped from the web without explicit licensing or informed consent.

The researchers who built those datasets did not respond to requests for comment.

How IBM is using the face database

IBM released its collection of annotated images to other researchers so that it can be used to develop "fairer" facial recognition systems. That means systems can more accurately identify people of all races, ages and genders.

"For the facial recognition systems to perform as desired, and the outcomes to become increasingly accurate, training data must be diverse and offer a breadth of coverage," said IBM's John Smith, in a blog post announcing the release of the data.

The dataset does not link the photos of people's faces to their names, which means any system trained to use the photos would not be able to identify named individuals. But civil liberty advocates and tech ethics researchers have still questioned the motives of IBM, which has a history of selling surveillance tools that have been criticized for infringing on civil liberties.

For example, in the wake of the 9/11 attacks, the company sold technology to the New York City police department that allowed it to search CCTV feeds for people with particular skin tones or hair color. IBM has also released an "intelligent video analytics" product that uses body camera surveillance to detect people by "ethnicity" tags, such as Asian, black or white.

IBM said in an email that the systems are "not inherently discriminatory," but added: "We believe that both the developers of these systems and the organizations deploying them have a responsibility to work actively to mitigate bias. It's the only way to ensure that AI systems will earn the trust of their users and the public. IBM fully accepts this responsibility and would not participate in work involving racial profiling."

Today, the company sells a system called IBM Watson Visual Recognition, which IBM says can estimate the age and gender of people depicted in images and, with the right training data, can be used by clients to identify specific people from photos or videos.

NBC News asked IBM what training data IBM Watson used for its commercial facial recognition abilities, pointing to a company blog post that stated that Watson is "transparent about who trains our AI systems, what data was used to train those systems." The company responded that it uses data "acquired from various sources" to train its AI models but does not disclose this data publicly to "protect our insights and intellectual property."

IBM said both in public statements and directly to NBC News that the Diversity in Faces dataset is purely for academic research and won't be used to improve the company's commercial facial recognition tools. This seems to conflict with the company's assertion in January in promotional materials that the release of the dataset is a direct response to research by MIT's Joy Buolamwini that showed that IBM's commercial facial recognition technology was much worse at accurately identifying darker-skinned women than lighter-skinned men.

When asked about this conflict, and particularly about how the Diversity in Faces dataset might have a real-world impact on reducing bias if IBM is not using it in commercial facial recognition products, Smith said in an email that the "scientific learnings on facial diversity will advance our understanding and allow us to create more fair and accurate systems in practice."

"We recognize that societal bias is not necessarily something we can fully tackle with science, but our aim is to address mathematical and algorithmic bias," Smith said.

Experts note that the distinction between the research wings and commercial operations of corporations such as IBM and Facebook is a blurry one. Ultimately, IBM owns any intellectual property developed by its research unit.

Even when algorithms are developed by academic researchers using noncommercial datasets, those algorithms are often later used by businesses, said Brian Brackeen, former CEO of the facial recognition company Kairos.

As an analogy, he said, "think of it as the money laundering of facial recognition. You are laundering the IP and privacy rights out of the faces."

IBM said it would not use the Diversity in Faces dataset in this way.

Photographers divided on IBM's dataset

An Austrian photographer and entrepreneur, Georg Holzer, uploaded his photos to Flickr to remember great moments with his family and friends, and he used Creative Commons licenses to allow nonprofits and artists to use his photos for free. He did not expect more than 700 of his images to be swept up to study facial recognition technology.

"I know about the harm such a technology can cause," he said over Skype, after NBC News told him his photos were in IBM's dataset. "Of course, you can never forget about the good uses of image recognition such as finding family pictures faster, but it can also be used to restrict fundamental rights and privacy. I can never approve or accept the widespread use of such a technology."

Holzer was concerned that a company like IBM - even its research division - had used photos he published under a noncommercial license.

"Since I assume that IBM is not a charitable organization and at the end of the day wants to make money with this technology, this is clearly a commercial use," he said.

Dolan Halbrook, who is based in Portland, Oregon, and has 452 photos in the dataset, agreed that IBM should have asked his permission.

"I'm annoyed at them being used without prior notification and a chance to review which ones would be included," Halbrook said. "I'm ambivalent about improving the technology itself."

Other photographers were glad to hear that their images may be used to advance the field of facial recognition.

"Facial recognition is one of those things we can't uninvent, so having a reliable system is better than one that generates errors and false identifications," said Neil Moralee, a food consultant and photographer based in the U.K. who specializes in portraits.

Guillaume Boppe from Switzerland agreed. "If the pictures of faces I shot are helping AI to improve, reducing false detection and ultimately improving global safety, I'm fine with it," he said.

Sebastian Gambolati, from Argentina, was happy to contribute to more accurate technology for finding missing people or tracking criminals, but he said it would have been "nice if they asked."

"In my Flickr account there are a lot of photos I took from people in events that aren't close to me," he said, "and I don't know what they think about the company using their photos without their consent."

Few options for opting out

IBM does offer an opt-out model of sorts: People can contact IBM with individual links to photographs they want removed from the dataset - either ones they have taken or ones they are featured in - and IBM says it will remove them, according to its privacy notice." However, there's no easy way to know if you are featured in the dataset, and even if you find out that you are, IBM said it will not remove photos on the basis of your Flickr user ID unless you also have links to each of the photographs.

When NBC News alerted one photographer, who asked not to be named for privacy reasons, that more than 1,000 of his photos were included in IBM's dataset, he tried to opt out by sending IBM his Flickr user ID. IBM told him that none of his photos were in the dataset, according to an email viewed by NBC News. When NBC News shared specific links to some of his photos in IBM's dataset, the company blamed an "indexing bug" for its initial inability to confirm that his images were included.After more than a week, IBM confirmed it had removed the four photos he had provided links to. According to NBC News' analysis, he still has 1,001 photos in the dataset.

Smith, of IBM, said in a statement that all URL removal requests had been completed.

Even once an image is removed from the IBM dataset, it won't be removed from the versions of the dataset already shared with research partners (about 250 organizations have requested access so far), IBM said. Nor will it be removed from the underlying Flickr dataset.

For those caught up in IBM's dataset or others like it, this makes the notion of opting out seem hopeless.

There may, however, be legal recourse in some jurisdictions thanks to the rise of privacy laws acknowledging the unique value of photos of people's faces. Under Europe's General Data Protection Regulation, photos are considered "sensitive personal information" if they are used to confirm an individual's identity. Residents of Europe who don't want their data included can ask IBM to delete it. If IBM doesn't comply, they can complain to their country's data protection authority, which, if the particular photos fall under the definition of "sensitive personal information," can levy fines against companies that violate the law.

In the U.S., some states have laws that could be relevant. Under the Illinois Biometric Information Privacy Act, for example, it can be a violation to capture, store and share biometric information without a person's written consent. According to the act, biometric information includes fingerprints, iris scans and face geometry.

"This is the type of mass collection and use of biometric data that can be easily abused, and appears to be taking place without the knowledge of those in the photos," said Jay Edelson, a Chicago-based class-action lawyer currently suing Facebook for its use of facial recognition tools.

So far neither of these laws has been rigorously tested.

IBM declined to comment on the laws.

'Incredibly oppressive the more accurate it gets'

Aside from the privacy issues, a bigger question remains: Are more accurate facial recognition systems actually "fairer"? And is it possible for facial recognition to be fair at all?

"You've really got a rock-and-a-hard-place situation happening here," said Woody Hartzog, a professor of law and computer science at Northeastern University. "Facial recognition can be incredibly harmful when it's inaccurate and incredibly oppressive the more accurate it gets."

While there are benign uses for facial recognition, it can also be used to surveil and target people of color and other vulnerable and minority communities. Facial recognition databases of mugshots are more likely to include African-Americans, Latinos and immigrants, since those groups are targeted by biased policing practices, civil rights groups say. This means that those people are far more "findable" by facial recognition technology, even if they were wrongly arrested when their mugshot was taken.

The use of facial recognition surveillance systems by law enforcement is so controversial that a coalition of more than 85 racial justice and civil rights groups have called for tech companies to refuse to sell the technology to governments. These groups argue that the technology exacerbates "historical and existing bias" that harms communities that are already "over-policed and over-surveilled."

"These systems are being deployed in oppressive contexts, often by law enforcement," said Whittaker, of the AI Now Institute, "and the goal of making them better able to surveil anyone is one we should look at very skeptically."

CORRECTION (March 12, 2019, 8:18 p.m. ET): An earlier version of this article incorrectly described Brian Brackeen's position at Kairos. He is the former CEO, not the current one.

# data[].attributes.attachment_link.title
Facial recognition's 'dirty little secret': Social media photos used without consent

# data[].attributes.attachment_link.description
Facial recognition can log you into your iPhone, track criminals through crowds and identify loyal customers in stores. The technology - which is imperfect but improving rapidly - is based on algorithms that learn how to recognize human faces and the hundreds of ways in which each one is unique.

# data[].attributes.attachment_link.title
This takedown of the Gillette ad backlash is a fascinating take on how the media works

# data[].attributes.attachment_link.description
You'll have seen how some people - okay, some blokes - were really, really angry about that Gillette #metoo advert which examined the issue of 'toxic masculinity'. People like Piers Morgan, for instance. And it became a proper news story about the company facing a backlash and even a boycott over the 2-minute ad.

# data[].attributes.headline
Tommy Robinson banned from Facebook and Instagram

# data[].attributes.subject
Tommy Robinson banned from Facebook and Instagram

# data[].attributes.attachment_link.readable_text
Tommy Robinson has been banned from Facebook and Instagram after repeatedly breaking hate speech policies.

Both pages were removed on Tuesday in response to "organised hate" and the targetting of Muslims.

"Tommy Robinson's Facebook page has repeatedly broken standards, posting material that uses dehumanising language and calls for violence targetted at Muslims," a Facebook spokesperson said.

"He has also behaved in ways that violate our policies around organised hate. As a result, in accordance with our policies, we have removed Tommy Robinson's official Facebook page and Instagram profile.

"This is not a decision we take lightly, but individuals and organisations that attack others on the basis of who they are have no place on Facebook or Instagram."

It comes after Facebook issued a final written warning to administrators of Robinson's account on 24 January, over posts including one "calling Muslims filthy scumbags" and "calling on people to 'make war' on Muslims".

The page was deleted because of further violations and support for "recognised hate figures" including Proud Boys founder Gavin McInnes - who Robinson is scheduled to tour Australia with next month - and the far-right group Generation Identity.

The anti-Islam activist has previously been removed from Twitter and Paypal over his activities, and his website was deleted in December.

Counter-extremist groups have long called for Robinson's Facebook page to be removed following a series of controversies.

The profile had a "donate" button until it was removed amid concern a tool originally intended for charities was being abused.

The social media company said it would take action "when ideas and opinions cross the line and amount to hate speech that may create an environment of intimidation and exclusion for certain groups in society - in some cases with potentially dangerous offline implications".

On the same day Robinson's page was deleted, several supporters reported that their accounts had been disabled, including former Breitbart London editor Raheem Kassam.

Robinson's Facebook page, which had more than 1 million followers, was his primary platform for communicating with fans and organising demonstrations.

He said the "corrupt media and the establishment" were trying "to silence any opposition to their globalist plans".

Supporters characterised the ban as a "free speech" crackdown, but it was welcomed by MPs including Harriet Harman and Tom Watson, who said the move came "far too late".

On Saturday, Robinson held a protest outside the BBC headquarters in Salford over an upcoming Panorama edition focused on him, and broadcast his own "documentary" on Facebook.

He used posts to ask for donations and take aim at issues and individuals, including politicians, journalists and members of the public.

Robinson posted videos doorstepping critics, with police being called after he visited the parents of a student who had been archiving his social media posts in December.

Last month, he posted a Facebook live from outside a Scottish National Party MP's surgery, and last May a broadcast from outside a grooming gang trial in Leeds saw him jailed.

Robinson, whose real name is Stephen Yaxley-Lennon, was freed in August over procedural failings and is awaiting a decision from the attorney general on whether a contempt of court case will proceed.

In November, he was appointed by Ukip leader Gerard Batten as the party's adviser on grooming gangs and prisons.

Mr Batten claimed Robinson's Facebook account was "shut down for BS reasons" and claimed the "left's establishment are closing ranks and counterattacking".

He has defended Robinson's appointment, but in a December Ukip Facebook video he admitted: "I do wonder if I made a wise decision, but I've asked him to do this for us and what he can bring to the table is access to a million Facebook followers."

In recent months, the English Defence League (EDL) founder has been encouraging fans to sign up for a mailing list in case he was "censored" by Facebook.

He retains a YouTube account, which has more than 288,000 subscribers, but had not seen activity in almost a year until Monday.

Robinson also has a profile on Gab, which is known for its far-right user base, but had not posted since last April.

Nick Lowles, chief executive of Hope not Hate, said: "Stephen Lennon is a far-right thug who uses his platform to bully, abuse and stir up division, monetising his hatred to earn huge sums while hiding behind a fake free speech mantle.

"Lennon has a long record of abuse towards minorities such as Muslims, so we welcome today's decision as well as Facebook's continued actions in cleaning up their platform."

Weyman Bennett, joint secretary of Unite Against Fascism, said: "This decision should have been taken much earlier. Robinson has a sordid history of instigating religious and racial hatred online."
We'll tell you what's true. You can form your own view.
At The Independent, no one tells us what to write. That's why, in an era of political lies and Brexit bias, more readers are turning to an independent source. Subscribe from just 15p a day for extra exclusives, events and ebooks - all with no ads.
Subscribe now

# data[].attributes.attachment_link.title
Tommy Robinson permanently banned from Facebook and Instagram

# data[].attributes.attachment_link.description
Tommy Robinson has been banned from Facebook and Instagram after repeatedly breaking hate speech policies.

# data[].attributes.headline
Security Flaws at the heart of the Panama papers

# data[].attributes.subject
Security Flaws at the heart of the Panama papers

# data[].attributes.attachment_link.readable_text
RODRIGO ARANGUA/AFP/Getty Images

The front-end computer systems of Mossack Fonseca are outdated and riddled with security flaws, analysis has revealed.

The law firm at the centre of the Panama Papers hack has shown an "astonishing" disregard for security, according to one expert. Amongst other lapses, Mossack Fonseca has failed to update its Outlook Web Access login since 2009 and not updated its client login portal since 2013.

Mossack Fonseca's client portal is also vulnerable to the DROWN attack, a security exploit that targets servers supporting the obsolete and insecure SSL v2 protocol. The portal, which runs on the Drupal open source CMS, was last updated in August 2013, according to the site's changelog.

On its main website Mossack Fonseca claims its Client Information Portal provides a "secure online account" allowing customers to access "corporate information anywhere and everywhere". The version of Drupal used by the portal has at least 25 vulnerabilities, including a high-risk SQL injection vulnerability that allows anyone to remotely execute arbitrary commands. Areas of the portal's backend can also be accessed by guessing the URL structure, a security researcher noted.

Read next

Mossack Fonseca's webmail system, which runs on Microsoft's Outlook Web Access, was last updated in 2009, while its main site runs a version of WordPress that is three months out of date. A further vulnerability makes it possible to easily access files uploaded to the backend of Mossack Fonseca's site simply by guessing the URL. "It shows the way they configured the server and the way they configured the website is not within the best security practices," an anonymous source told WIRED. They continued to say that the method could be used by other people to access the data. "We're talking about a misconfigured server that enables directory listings."

They seem to have been caught in a time warp. If I were a client of theirs I'd be very concerned that they were communicating using such outdated technology
Alan Woodward, Surrey University

Professor Alan Woodward, a computer security expert from Surrey University told WIRED that Mossack Fonseca's front end seemed "horribly" out of date. "I can't understand this," Woodward continued. "Take something like Outlook Web Access - if you keep your Exchange Server up to date this just comes along naturally. They seem to have been caught in a time warp. If I were a client of theirs I'd be very concerned that they were communicating using such outdated technology."

Mossack Fonseca's emails were also not encrypted, according to privacy expert Christopher Soghoian who noted the company did not use the TLS security protocol. "Given the business they're in, I find it quite surprising that they haven't thought about securing their emails better," Angela Sasse, professor of human-centred technology at University College London, told WIRED.

Read next

"I would regard TLS encryption as okay for a not very high risk organisation, if it is done properly and looked after. The awareness of the risk and how easily these services can be attacked seems to not have been there."

Precisely what vulnerability the attacker used is not known and Mossack Fonseca has said it is carrying out "an in-depth investigation with experts", while also taking "additional measures" to strengthen its systems. In a leaked email to customers Mossack Fonseca confirmed an "unauthorised breach" of its email servers. Company partner Ramon Fonseca has since said the leak was not "an inside job" and that the company had been hacked by servers based abroad. The company did not respond to requests for comment.

This points at an insider with enough access privileges to get to see all the data, but not enough privileges to be able to copy it all quickly to one disc
Eerke Boiten, University of Kent

It also remains unclear who carried out the attacks. University of Kent senior computing lecturer Eerke Boiten told WIRED that the leak may in fact be the work of an insider. "We do know that it was a lot of data, and that it came out gradually," he said. "This points at an insider with enough access privileges to get to see all the data, but not enough privileges to be able to copy it all quickly to one disc."

Woodward disagreed, saying the vulnerabilities in Mossack Fonseca's systems made it "vulnerable to external scanning and exploitation". The attacker may even have been a nation state, he continued. "If I were a betting man I would place a two way bet between an external hacker who got lucky by probing, was shocked by what they saw and leaked it, and a nation state fed up with tax avoidance."

RODRIGO ARANGUA/AFP/Getty Images

What little is known about the source of the leak comes from details published by German newspaper Suddeutsche Zeitung. Communicating via encrypted chat in late 2014, the source warned his or her life was "in danger" but that they had data from law firm Mossack Fonseca that they wanted to share. When asked how much data they had, the source replied "more than you have ever seen," according to the newspaper.

It took almost a year for all the data to arrive, with the source sending it in dribs and drabs. Dating back to the 1970s, the 11.5 million documents - the biggest leak in history - total 2.6 terabytes.

I knew one law firm who, as part of their so-called governance, walked around the building in the summer to make sure papers hadn't blown out (the window)
Dr Daniel Dresner, Manchester University

The Panama Papers detail 214,488 offshore entities related to public officials held by Mossack Fonseca. The leak includes emails, contracts, scanned documents and transcripts. Broken down by file type, the leak comprises 4.8 million emails, three million database files, 2.1 million PDFs, 1.1 million images, 320,166 text files and 2,242 files in other formats. All the files came organised in folders for the individual shell firms they related to. A full list of companies and people linked to the offshore entities will be published in May 2016.

Dr Daniel Dresner, a lecturer in cyber security at Manchester University's school of computer science, told WIRED that Mossack Fonseca's seemingly lax security protocols were not unusual amongst law firms. "There's always a feeling in the legal fraternity that whatever happens they'll be able to get off the wrap because they're clever legal people," he said. "People are now starting to realise that legal companies are a great target. When you think about the size of stuff that they're negotiating, who they're negotiating for and the number of different parties involved, the motivation is there for people who want a bit of insider information."

Dresner added that the poor data protection practices of some law firms offline had clearly been duplicated online. "Look at the back seat of the car of the average partner's BMW and I think you'd be quite shocked. These guys still take large bundles of papers around tied up with ribbons," he said. "I knew one law firm who, as part of their so-called governance, walked around the building in the summer to make sure papers hadn't blown out [the window]." "There are plenty of vectors," Dresner said when presented with a list of potential holes in Mossack Fonseca's systems. "But they've put a great big invitation out there with an RSVP on it."

# data[].attributes.attachment_link.title
The security flaws at the heart of the Panama Papers

# data[].attributes.attachment_link.description
The front-end computer systems of Mossack Fonseca are outdated and riddled with security flaws, analysis has revealed. The law firm at the centre of the Panama Papers hack has shown an "astonishing" disregard for security, according to one expert.

# data[].attributes.attachment_link.readable_text
Warning: NSFW language throughout

The SNP's Mhairi Black, the 20-year-old student who beat shadow foreign secretary Douglas Alexander in Paisley and Renfrewshire South, will soon arrive in Westminster as the country's youngest MP since 1667.

While that's quite the mantle to take up - especially when you haven't finished university yet - Black is clearly still a pretty normal young 'un, as tweets since the age of 14 show. Here are some of our favourite Mhairi Black posts:

"Woke up beside half a can of Tennents and a full pizza and more money than I came out with. I call that a success!"

Smashing the incumbent Labour candidate by winning 51 per cent of the vote was the only possibly way forward from here.

"Smirnoff Ice is the drink of gods - I cannae handle this c--- man!"

Teenage kicks really are so hard to beat.

"Celtic, yer a joke!#scum"

Her footballing sympathies are not exactly One Scotland but passion in a politician is always good.

Oasis were formed in 1991. Black was born in 1994. Now we feel old.

PMQs could get a wee bit more colourful with Black on the benches...

More: [Six contenders to be the next Labour leader] 1

# data[].attributes.attachment_link.title
We enjoyed reading Mhairi Black's NSFW tweets as a teenager

# data[].attributes.attachment_link.description
Warning: NSFW language throughout The SNP's Mhairi Black, the 20-year-old student who beat shadow foreign secretary Douglas Alexander in Paisley and Renfrewshire South, will soon arrive in Westminster as the country's youngest MP since 1667.

# data[].attributes.attachment_link.description
An empowering first step for staff and students to reflect on and develop their digital capabilities

# data[].attributes.attachment_link.description
Ofcom's annual Internet Use and Attitudes report is published as part of our media literacy duties.

# data[].attributes.attachment_link.title
The anti-EU lies are back to exploit Britain's weak spot again | Polly Toynbee

# data[].attributes.attachment_link.description
A campaign of fake news about the EU has gone viral. We're only vulnerable to this because of 40 years of weak leadership, says Guardian columnist Polly Toynbee

# data[].attributes.attachment_link.readable_text
The data analytics firm that worked with Donald Trump's election team and the winning Brexit campaign harvested millions of Facebook profiles of US voters, in one of the tech giant's biggest ever data breaches, and used them to build a powerful software program to predict and influence choices at the ballot box.

A whistleblower has revealed to the how Cambridge Analytica - a company owned by the hedge fund billionaire Robert Mercer, and headed at the time by Trump's key adviser Steve Bannon - used personal information taken without authorisation in early 2014 to build a system that could profile individual US voters, in order to target them with personalised political advertisements.

Christopher Wylie, who worked with a Cambridge University academic to obtain the data, told the Observer: "We exploited Facebook to harvest millions of people's profiles. And built models to exploit what we knew about them and target their inner demons. That was the basis the entire company was built on."

Documents seen by the Observer, and confirmed by a Facebook statement, show that by late 2015 the company had found out that information had been harvested on an unprecedented scale. However, at the time it failed to alert users and took only limited steps to recover and secure the private information of more than 50 million individuals.

Profile
Cambridge Analytica: the key players

Alexander Nix, CEO
An Old Etonian with a degree from Manchester University, Nix, 42, worked as a financial analyst in Mexico and the UK before joining SCL, a strategic communications firm, in 2003. From 2007 he took over the company's elections division, and claims to have worked on 260 campaigns globally. He set up Cambridge Analytica to work in America, with investment from Robert Mercer.
Aleksandr Kogan, data miner
Aleksandr Kogan was born in Moldova and lived in Moscow until the age of seven, then moved with his family to the US, where he became a naturalised citizen. He studied at the University of California, Berkeley, and got his PhD at the University of Hong Kong before joining Cambridge as a lecturer in psychology and expert in social media psychometrics. He set up Global Science Research (GSR) to carry out CA's data research. While at Cambridge he accepted a position at St Petersburg State University, and also took Russian government grants for research. He changed his name to Spectre when he married, but later reverted to Kogan.
Steve Bannon, former board member
A former investment banker turned "alt-right" media svengali, Steve Bannon was boss at website Breitbart when he met Christopher Wylie and Nix and advised Robert Mercer to invest in political data research by setting up CA. In August 2016 he became Donald Trump's campaign CEO. Bannon encouraged the reality TV star to embrace the "populist, economic nationalist" agenda that would carry him into the White House. That earned Bannon the post of chief strategist to the president and for a while he was arguably the second most powerful man in America. By August 2017 his relationship with Trump had soured and he was out.

Robert Mercer, investor
Robert Mercer, 71, is a computer scientist and hedge fund billionaire, who used his fortune to become one of the most influential men in US politics as a top Republican donor. An AI expert, he made a fortune with quantitative trading pioneers Renaissance Technologies, then built a $60m war chest to back conservative causes by using an offshore investment vehicle to avoid US tax.

Rebekah Mercer, investor
Rebekah Mercer has a maths degree from Stanford, and worked as a trader, but her influence comes primarily from her father's billions. The fortysomething, the second of Mercer's three daughters, heads up the family foundation which channels money to rightwing groups. The conservative mega‑donors backed Breitbart, Bannon and, most influentially, poured millions into Trump's presidential campaign.

Thank you for your feedback.

The New York Times is reporting that copies of the data harvested for Cambridge Analytica could still be found online; its reporting team had viewed some of the raw data.

The data was collected through an app called thisisyourdigitallife, built by academic Aleksandr Kogan, separately from his work at Cambridge University. Through his company Global Science Research (GSR), in collaboration with Cambridge Analytica, hundreds of thousands ofusers were paid to take a personality test and agreed to have their data collected for academic use.

However, the app also collected the information of the test-takers' Facebook friends, leading to the accumulation of a data pool tens of millions-strong. Facebook's "platform policy" allowed only collection of friends' data to improve user experience in the app and barred it being sold on or used for advertising. The discovery of the unprecedented data harvesting, and the use to which it was put, raises urgent new questions about Facebook's role in targeting voters in the US presidential election. It comes only weeks after indictments of 13 Russians by the special counsel Robert Mueller which stated they had used the platform to perpetrate "information warfare" against the US.

Cambridge Analytica and Facebook are one focus of an inquiry into data and politics by the British Information Commissioner's Office. Separately, the Electoral Commission is also investigating what role Cambridge Analytica played in the EU referendum.

"We are investigating the circumstances in which Facebook data may have been illegally acquired and used," said the information commissioner Elizabeth Denham. "It's part of our ongoing investigation into the use of data analytics for political purposes which was launched to consider how political parties and campaigns, data analytics companies and social media platforms in the UK are using and analysing people's personal information to micro-target voters."

On Friday, four days after the Observer sought comment for this story, but more than two years after the data breach was first reported, Facebook announced that it was suspending Cambridge Analytica and Kogan from the platform, pending further information over misuse of data. Separately, Facebook's external lawyers warned the Observer it was making "false and defamatory" allegations, and reserved Facebook's legal position.

Key Trump adviser Steve Bannon Photograph: Alain Robert/Sipa/Rex/Shutterstock

The revelations provoked widespread outrage. The Massachusetts Attorney General Maura Healey announced that the state would be launching an investigation. "Residents deserve answers immediately from Facebook and Cambridge Analytica," she said on Twitter.

The Democratic senator Mark Warner said the harvesting of data on such a vast scale for political targeting underlined the need for Congress to improve controls. He has proposed an Honest Ads Act to regulate online political advertising the same way as television, radio and print. "This story is more evidence that the online political advertising market is essentially the Wild West. Whether it's allowing Russians to purchase political ads, or extensive micro-targeting based on ill-gotten user data, it's clear that, left unregulated, this market will continue to be prone to deception and lacking in transparency," he said.

Last month both Facebook and the CEO of Cambridge Analytica, Alexander Nix, told a parliamentary inquiry on fake news: that the company did not have or use private Facebook data.

Simon Milner, Facebook's UK policy director, when asked if Cambridge Analytica had Facebook data, told MPs: "They may have lots of data but it will not be Facebook user data. It may be data about people who are on Facebook that they have gathered themselves, but it is not data that we have provided."

Cambridge Analytica's chief executive, Alexander Nix, told the inquiry: "We do not work with Facebook data and we do not have Facebook data."

Wylie, a Canadian data analytics expert who worked with Cambridge Analytica and Kogan to devise and implement the scheme, showed a dossier of evidence about the data misuse to the Observer which appears to raise questions about their testimony. He has passed it to the National Crime Agency's cybercrime unit and the Information Commissioner's Office. It includes emails, invoices, contracts and bank transfers that reveal more than 50 million profiles - mostly belonging to registered US voters - were harvested from the site in one of the largest-ever breaches of Facebook data. Facebook on Friday said that it was also suspending Wylie from accessing the platform while it carried out its investigation, despite his role as a whistleblower.

At the time of the data breach, Wylie was a Cambridge Analytica employee, but Facebook described him as working for Eunoia Technologies, a firm he set up on his own after leaving his former employer in late 2014.

The evidence Wylie supplied to UK and US authorities includes a letter from Facebook's own lawyers sent to him in August 2016, asking him to destroy any data he held that had been collected by GSR, the company set up by Kogan to harvest the profiles.

That legal letter was sent several months after the Guardian first reported the breach and days before it was officially announced that Bannon was taking over as campaign manager for Trump and bringing Cambridge Analytica with him.

"Because this data was obtained and used without permission, and because GSR was not authorised to share or sell it to you, it cannot be used legitimately in the future and must be deleted immediately," the letter said.

Facebook did not pursue a response when the letter initially went unanswered for weeks because Wylie was travelling, nor did it follow up with forensic checks on his computers or storage, he said.

"That to me was the most astonishing thing. They waited two years and did absolutely nothing to check that the data was deleted. All they asked me to do was tick a box on a form and post it back."

Paul-Olivier Dehaye, a data protection specialist, who spearheaded the investigative efforts into the tech giant, said: "Facebook has denied and denied and denied this. It has misled MPs and congressional investigators and it's failed in its duties to respect the law.

"It has a legal obligation to inform regulators and individuals about this data breach, and it hasn't. It's failed time and time again to be open and transparent."

We exploited Facebook to harvest millions of profiles. And built models to exploit that and target their inner demons

Christopher Wylie

A majority of American states have laws requiring notification in some cases of data breach, including California, where Facebook is based.

Facebook denies that the harvesting of tens of millions of profiles by GSR and Cambridge Analytica was a data breach. It said in a statement that Kogan "gained access to this information in a legitimate way and through the proper channels" but "did not subsequently abide by our rules" because he passed the information on to third parties.

Facebook said it removed the app in 2015 and required certification from everyone with copies that the data had been destroyed, although the letter to Wylie did not arrive until the second half of 2016. "We are committed to vigorously enforcing our policies to protect people's information. We will take whatever steps are required to see that this happens," Paul Grewal, Facebook's vice-president, said in a statement. The company is now investigating reports that not all data had been deleted.

Kogan, who has previously unreported links to a Russian university and took Russian grants for research, had a licence from Facebook to collect profile data, but it was for research purposes only. So when he hoovered up information for the commercial venture, he was violating the company's terms. Kogan maintains everything he did was legal, and says he had a "close working relationship" with Facebook, which had granted him permission for his apps.

Quick Guide
How the Cambridge Analytica story unfolded

In December 2016, while researching the US presidential election, Carole Cadwalladr came across data analytics company Cambridge Analytica, whose secretive manner and chequered track record belied its bland, academic-sounding name.

Her initial investigations uncovered the role of US billionaire Robert Mercer in the US election campaign: his strategic "war" on mainstream media and his political campaign funding, some apparently linked to Brexit.

She found the first indications that Cambridge Analytica might have used data processing methods that breached the Data Protection Act. That article prompted Britain's Electoral Commission and the Information Commissioner's Office to launch investigations whose remits include Cambridge Analytica's use of data and its possible links to the EU referendum. These investigations are continuing, as is a wider ICO inquiry into the use of data in politics.

While chasing the details and ramifications of complex manipulation of both data and funding law, Cadwalladr came under increasing attacks, both online and professionally, from key players.

The Leave.EU campaign tweeted a doctored video that showed her being violently assaulted, and the Russian embassy wrote to the Observer to complain that her reporting was a "textbook example of bad journalism".

But the growing profile of her reports also gave whistleblowers confidence that they could trust her to not only understand their stories, but retell them clearly for a wide audience.

Her network of sources and contacts grew to include not only former employees who regretted their work but academics, lawyers and others concerned about the impact on democracy of tactics employed by Cambridge Analytica and associates.

Cambridge Analytica is now the subject of special prosecutor Robert Mueller's probing of the company's role in Donald Trump's presidential election campaign. Investigations in the UK remain live.

Thank you for your feedback.

The Observer has seen a contract dated 4 June 2014, which confirms SCL, an affiliate of Cambridge Analytica, entered into a commercial arrangement with GSR, entirely premised on harvesting and processing Facebook data. Cambridge Analytica spent nearly $1m on data collection, which yielded more than 50 million individual profiles that could be matched to electoral rolls. It then used the test results and Facebook data to build an algorithm that could analyse individual Facebook profiles and determine personality traits linked to voting behaviour.

The algorithm and database together made a powerful political tool. It allowed a campaign to identify possible swing voters and craft messages more likely to resonate.

"The ultimate product of the training set is creating a 'gold standard' of understanding personality from Facebook profile information," the contract specifies. It promises to create a database of 2 million "matched" profiles, identifiable and tied to electoral registers, across 11 states, but with room to expand much further.

At the time, more than 50 million profiles represented around a third of active North American Facebook users, and nearly a quarter of potential US voters. Yet when asked by MPs if any of his firm's data had come from GSR, Nix said: "We had a relationship with GSR. They did some research for us back in 2014. That research proved to be fruitless and so the answer is no."

Cambridge Analytica said that its contract with GSR stipulated that Kogan should seek informed consent for data collection and it had no reason to believe he would not.

GSR was "led by a seemingly reputable academic at an internationally renowned institution who made explicit contractual commitments to us regarding its legal authority to license data to SCL Elections", a company spokesman said.

SCL Elections, an affiliate, worked with Facebook over the period to ensure it was satisfied no terms had been "knowingly breached" and provided a signed statement that all data and derivatives had been deleted, he said. Cambridge Analytica also said none of the data was used in the 2016 presidential election.

Steve Bannon's lawyer said he had no comment because his client "knows nothing about the claims being asserted". He added: "The first Mr Bannon heard of these reports was from media inquiries in the past few days." He directed inquires to Nix.

The Observer

The Observer is the world's oldest Sunday newspaper, founded in 1791. It is published by Guardian News & Media and is editorially independent.

# data[].attributes.attachment_link.title
Revealed: 50 million Facebook profiles harvested for Cambridge Analytica in major data breach

# data[].attributes.attachment_link.description
Whistleblower describes how firm linked to former Trump adviser Steve Bannon compiled user data to target American voters

# data[].attributes.attachment_link.readable_text
I am disappointed every time I guest lecture a law school class.

Because anecdote is often more compelling than data, I'll start with an example from two weeks ago. An adjunct professor who teaches one of those great law school classes with cool titles like Tomorrow's Lawyer had his students take the Word module of my They performed exactly as you (well, I) would expect of an untrained group: poorly. Their anonymized scores are below:

I'll explain the scoring in a subsequent post. But, for now, just focus on the accuracy. On average, students were able to correctly complete less than a third of the following tasks in a live Word document:

Accept/Turn-off changes and comments

Cut & Paste

Replace text

Format text (font, margin)

Footers

Insert hyperlink

Apply/Modify style

Insert/Update cross-references

Insert page break

Insert non-breaking space

Clean document properties

Create comparison document

To put it in anecdotal context (though these numbers are representative of the larger data set), around the same time the class was taking the assessment, a similarly-sized pilot group at a large law firm achieved an average accuracy of 68% while a group of lawyers and staff who had just been through LTA-specific training achieved an average accuracy of 95%. Training matters .
Training matters even for so-called digital natives like the law school students I've tested. Acquiring a Twitter account in utero does not engender natural facility with technological tools because most technological tools are not intuitive.

My wife thinks my sons are geniuses. One piece of evidence she submits in their favor is how well they use an iPad. I agree with her that the fact that my 1.5 year old can use an iPad is a testament to genius. But not his. For me, it is a testament to the genius of the designers at Apple who created a device so intuitive that 1.5 year old can use it. The kid touches a picture, it moves. Congratulations to him!

While we can discuss design principles that would move work software in the direction of our consumer experience, I don't actually believe that individual apps are the best basis of comparison for complex business software. Rather than thinking of Word as an app, we should think of it as a bundle of apps. Each of the icons on the ribbon is a solution to a particular problem. The challenge is that there are so many icons (just as there are so many apps). While design principles can bridge some of the gap, there remains a tradeoff between depth and intuitiveness. Most of us therefore only become comfortable with a few functions while ignoring the rest (just as most of us use a limited number of mostly single-purpose apps):

In this, the younger generation is no different than their predecessors. Survival is the threshold they achieve with most of their technology. That their milieu demands facility with a few more social apps sooner does not change the fundamental fact that using technology properly is a collection of acquired skills, not some innate talent that Lamarckian evolution bestowed on those under the age of 30. Expecting them to automatically know how to use complicated technology because of their familiarity with basic technology is like expecting them to automatically know how to prepare a gourmet meal because they know how to cook a Hot Pocket in the microwave.

Digital native first entered the popular lexicon in 2001. In an article entitled Digital Natives, Digital Immigrants , education consultant Marc Prensky explained that "Our students today are all 'native speakers' of the digital language of computers, video games and the Internet." In many respects, the article looks terrible 14 years later. Prensky, for example, speaks glowingly of Digital Natives' ability to multi-task, despite the fact that contemporary research (subsequently extended and validated ) had already demonstrated the opposite. But just because it suffers from some poor assumptions and hyperbole (Prensky claims we've already arrived at a singularity ) does not mean the thesis needs to be totally rejected.

You can reconcile (i) a belief that, on average, younger generations are more accustomed to technology (and the rapid evolution thereof) than their predecessors with (ii) the recognition that this comfort does not automatically translate into proficiency. Indeed, in his seminal article, Prensky talks about the need to train a bunch of Digital Natives on a CAD program that "contained hundreds of new buttons, options and approaches." Prensky takes pride in efforts to gamify the training and "create a series of graded tasks into which the skills to be learned were embedded." For Prensky, this approach translated training into the language of the Digital Native. He reports that the main impediment was the reluctance of the Digital Immigrant professors to adjust their pedagogical approach. But even Prensky realized that the Digital Natives still needed training ( competence-based assessments paired with synchronous, active learning ).

Prensky's article was anecdotal, not empirical. It did not address the fact that most of what Digital Natives did with technology was related to consumption, not application. It also ignored the inconvenient fact that it was Digital Immigrants who had invented the technologies on which the Digital Natives relied. And it introduced a term that conflates general familiarity with specific facility. A decade later, the London School of Economics would publish a paper entitled Digital natives: where is the evidence? The paper concluded that there was no real evidence of fundamental differences between generations. What differences existed were best explained by "breadth of use, experience, self-efficacy and education." Or as another academic paper would find:

Young people's engagements with digital technologies are varied and often unspectacular - in stark contrast to popular portrayals of the digital native. As such, the paper highlights a misplaced technological and biological determinism that underpins current portrayals of children, young people and digital technology.

Subsequently, the Organization of Economically Developed Countries (OECD) ran an international . Instead of asking people about their general comfort with technology, the study asked them to actually solve basic problems using technology (again, a competence-based assessment). Millennials did not fare well . In the words of , "U.S. millennials performed horribly." Or, as summarized it:

We hear about the superior tech savvy of people born after 1980 so often that we tend to assume it must be true. But is it?

.... It turns out, says a new report, that Millennials in the U.S. fall short when it comes to the skills employers want most: literacy (including the ability to follow simple instructions), practical math, and - hold on to your hat - a category called "problem-solving in technology-rich environments."

The advocacy group Change the Equation put out a related report about the High Cost of Low Technology Skills , which included the following graphics:

Digital natives are not at fault for the fact that comfort does not automatically translate into skill. The myths surrounding the digital native, however, have done them a disservice. The belief that they already know everything about technology has convinced us and them that they do not need training in technology. As discussed in the , the notion that they are already tech savvy introduces barriers associated with metacognition and mindset.

Metacognition is thinking about thinking. The related concept of metaignorance is ignorance about our own ignorance. We don't know what we don't know. Not recognizing how incompetent we are results in unfounded confidence in our own skill level. For this reason, the people most in need of training are the least likely to recognize it. Moreover, confidence begets ego. Those who have a high opinion of themselves are the least inclined to admit facts that undermine their self image.

The problems of ego are compounded by mindset. People with a fixed mindset believe that cognitive traits are stable. You are either smart or you aren't in the same way that you are either tall or you aren't. You are born with it, or you are not. To the extent tech acumen is treated as product of age, it will be approached with a fixed mindset, which means that people will try to hide their deficiencies rather than recognize and remedy them.

Because we take a fixed mindset approach, we tend to act as if neither older professionals nor younger professionals will benefit from training. The former because they lack the capacity. The latter because they lack the need. We are wrong on both counts.

I am not trying to pick on anyone. I understand why both older and younger generations buy into the myths surrounding digital natives. But they are myths with pernicious consequences. Many of the decisions about training (or lack thereof) made at law schools and legal employers rest on an illusory foundation. It isn't always what we don't know that gives us trouble, it's often what we know that ain't so.

Part 2 (originally posted separately)

90% of people don't know how to use CTRL+F to find a word in a document or web page . Instead, they search the old-fashioned way, manually skimming the text.

This preponderance of ignorance is stupefying to me. But I want to be very clear that I am using the word "ignorance" in its most neutral form--i.e., lack of information or knowledge--rather than to convey any judgment or pejorative connotation. Ignorance is unavoidable. The only settled part of the debate as to who was the last person to know everything is that the person is long dead.

The curse of ignorance is that you don't know what you don't know. have touched on this obstacle of metacognition, and our ignorance of our own ignorance. But there is another side of the coin: the curse of knowledge. The curse of knowledge is that once we know something, it is really hard to imagine not knowing it. This incapacity undermines communication and, especially, instruction because of the lack of shared information and assumptions. If I, for example, were going to put together some tips on internet research, I doubt that, absent the article cited above, I would have thought to include CTRL+F. I would have assumed that most everyone already knew it. I would have been wrong.

Indeed, I am a posterchild for both curses. I've told the many times that my inflection point in using technology involved a client discovering that I printed and scanned to create PDFs. But how was I supposed to know what I didn't know--there's an app for that--without already knowing it? Yet, several years later, I delegated a task where one of the steps involved converting a large volume of documents into PDF. I was shocked (shocked!!!) to find that the person was spending hours scanning and printing. I assumed that because I knew how to convert a file to PDF, they knew it, too, despite the fact that I had been Exhibit A that this was not knowledge everyone possessed.

Thus, whether we know something or not, we too often assume that others know it. The tech-averse frequently fall into the trap of thinking the tech-comfortable know everything there is to know about tech (i.e., they can't tell the difference between someone who knows slight more than them and someone who knows infinitely more than them). And those who know tech sometimes assume that their tech knowledge is widely shared.

Both curses are reason that competence-based assessments are such excellent training tools. Figuring out what people do and do not know is superior to speculation. But assessments alone are not enough. The primary objective of identifying gaps is to tailor the training to fill them. In this regard, I have been an abject failure in speaking to law school classes .

I speak to law school classes for free. I provide them a copy of my Legal Technology Assessment ("LTA") for free. I then provide a copy of the LTA Training Edition (which pairs the competence-based assessment with synchronous, active learning ) for free. Finally, they can retake the LTA (for free). Not only do have the opportunity to address identified deficiencies in their skill set but a qualifying score is also something they can add to the bottom of their resume to replace the meaningless "proficient in MS Office." After speaking to hundreds upon hundreds of students, I've had exactly zero take me up on my full offer.

The class I wrote about last week is representative. Twelve students took the LTA because it was a class assignment. The results (below) were bad, as usual. I spoke to them for 40 minutes and offered the Training Edition to anyone who wanted it. Only two of the twelve emailed to ask for the Training Edition. And, if history is any guide, neither of them will return to take and pass the LTA.

In approaching these classes, my idea is that taking the LTA beforehand will puncture delusions of adequacy. We won't get bogged down in an abstract conversation about how fluent they are with technology. 32% correct on some fairly simple Word tasks leaves little room for debate:

Pretty bad but not unexpected. As I try to communicate to them, it is not their fault. Everyone just that they know things that they had no way of knowing absent training. They are not stupid, lazy, or untalented. They are smart, hard working, and full of promise. They simply lack training in one particular area that has the potential to make their lives better.

On the issue of their immediate future, I point out that their most recent predecessors are miserable human beings. In fact, the students are auditioning for the unhappiest job in America .

I then try to persuade them that technology plays a role in this dissatisfaction. Before technology takes our jobs , it can make them easier. At least, in theory. The technology has to actually be good, and we have to use it correctly. Otherwise, it is a source of frustration rather than leverage. Technology initially substitutes for labor at the most severe pain points. Machines can reduce the hours spent reviewing, proofing, conforming, collating, updating, and otherwise fiddling around the edges of the substantive legal work. Using technology well can improve both speed and accuracy, as I try to convey in the video below, and thereby alleviate a fair amount of the agony associated with being a young lawyer:

One unsung key to retention could be technology. We found that technology training overall satisfaction of midlevel associates, as measured on our survey, was strongly statistically correlated to their law firm's scores on four questions involving technology. (The questions ask respondents to rate their firms' technology generally, as well as , support and use of technology in meeting client needs.)

.... In fact, eight of the top 11 firms in the national satisfaction rankings also were at the top on the technology questions. Conversely, many of the firms that occupy the bottom of the national satisfaction rankings also place low in the technology survey.

The AmLaw conclusions comport with an earlier study I cite from the National Conference of Bar Examiners that surveyed recent law graduates about the most important skills for young lawyers . Out of 30 skills, using basic office technology ranked 6th:

# data[].attributes.attachment_link.description
I am disappointed every time I guest lecture a law school class. Because anecdote is often more compelling than data, I'll start with an example from two weeks ago. An adjunct professor who teaches one of those great law school classes with cool titles like Tomorrow's Lawyer had his students take the Word module of my They performed exactly as you (well, I) would expect of an untrained group: poorly.

# data[].attributes.attachment_link.readable_text
It was 9 o'clock on a Sunday night last July when a journalist called Brian Krebs came upon the scoop of his life. The 42-year-old was at home in Virginia at the time, and wearing pyjamas. For years Krebs had written a popular blog about internet security, analysing thefts of consumer data from big companies around the world, Tesco, Adobe, Domino's Pizza among them. Now Krebs, as his weekend came to an end, was being tipped off about a more sensational breach. An anonymous informant had emailed him a list of links, directing him to caches of data that had been stolen from servers at a Canadian firm called Avid Life Media (ALM). Krebs vaguely knew of ALM. For years it had run a notorious, widely publicised web service called Ashley Madison, a dating site founded in 2008 with the explicit intention of helping married people have affairs with each other. "Life is short. Have an affair" was the slogan Ashley Madison used.

At the time Krebs received his tip-off, Ashley Madison claimed to have an international membership of 37.6 million, all of them assured that their use of this service would be "anonymous", "100% discreet". Only now Krebs was looking at the real names and the real credit-card numbers of Ashley Madison members. He was looking at street addresses and postcodes. Among documents in the leaked cache, Krebs found a list of telephone numbers for senior executives at ALM and Ashley Madison. He even found the personal mobile number of the CEO, a Canadian called Noel Biderman.

Some people found they could be identified by their height, weight, and erotic desires

"How you doing?" Krebs asked Biderman when he dialled and got through - still not sure, until this moment, that he was on to a legitimate story.

Biderman said: "You can probably guess."

Then the CEO of Ashley Madison began the slow, careful work of begging Krebs not to publish anything about the most appallingly intimate internet leak of the modern age.

Only a few hours later, in the west of England, a contentedly married man we'll call Michael woke up and went through his usual Monday-morning routine. Coffee. Email. A skim of the news online. Already Krebs's story about a hack of servers at Ashley Madison had been picked up by prominent media agencies. The story was a lead item on every news page Michael browsed. Infidelity site hacked, he read; a group calling itself the Impact Team claiming responsibility and threatening to release a full database of Ashley Madison customers, present and past, inside a month. More than 30 million people in more than 40 countries affected.

Though in the days to come the number of active users of Ashley Madison's service would be disputed - was that figure of 37.6 million for real? - Michael could say for sure there were many authentic adulterers who used the site because he was one of them. "I'd taken some elementary precautions," Michael told me recently, explaining that he'd registered on Ashley Madison with a secret email address and chosen a username by which he couldn't be personally identified. He had uploaded a photograph. He was experienced enough with adultery websites - Ashley Madison and a British equivalent called Illicit Encounters - to know that "if you don't put a photo up you won't get many responses". But the picture he chose was small and he was wearing sunglasses in it. "Deniable," Michael said.

Whenever he visited the site he was careful. If he wanted to log on to Ashley Madison to speak to women he would only do so on a work laptop he kept in his office at home. Michael had six internet browsers installed on the laptop, and one of these browsers could only be loaded via external hard drive - this was the browser he used to arrange affairs. So Michael was "irritated and surprised" to realise, that Monday morning, that his elaborate precautions had been pointless. He tried to work out ways in which he would be exposed if the hackers went through with their threat to release Ashley Madison's customer database.

Facebook Twitter Pinterest

Getting in on the act: moral crusaders, operating with impunity, began to shame and squeeze the exposed. Photograph: Carl Court/Getty Images

Subscriptions to the site were arranged so that women could use the service for free while men paid a monthly fee - this, in theory, to encourage an even balance in its membership. Michael had joined Ashley Madison after seeing it written about in a newspaper. He recalled getting a deal as a new signee and being charged something like £20 for his first month. He paid using his credit card. The profile name and email address he'd chosen were no threat, the photograph deniable - "but your credit card," Michael realised, "is your credit card." At this time there would have been a lot of men (even conservative estimates put the number of paid- up Ashley Madison subscribers at the time well into the millions) thinking: your credit card is your credit card.

Michael followed it all from his home computer as the story evolved, through July and into August, into an enormous, consistently strange, consistently ghastly global calamity.

On 18 August, Ashley Madison's entire customer database was indeed put online. In the subsequent panic, rewards for information about the hackers were offered. Police in Toronto (the city where ALM was based) vowed to find the culprits. Meanwhile politicians, priests, military members, civil servants, celebrities - these and hundreds of other public figures were found among the listed membership. Millions more, formerly anonymous, suddenly had their private details sprayed out on to the internet. It varied according to an individual's caution when signing up to the site, and to their luck, and to their gender (the men in general more exposed because of Ashley Madison's requirement they pay by credit card), but after the leak some people found they could be identified not only by their names and their addresses but also by their height, their weight, even their erotic preferences.

Moral crusaders, operating with impunity, began to shame and squeeze the exposed. In Alabama editors at a newspaper decided to print in its pages all the names of people from the region who appeared on Ashley Madison's database. After some high-profile resignations all around North America, people wondered if there might not be a risk of more tragic repercussions. Brian Krebs, with some prescience, wrote a blog advising sensitivity: "There's a very real chance that people are going to overreact," he wrote. "I wouldn't be surprised if we saw people taking their lives because of this."

A small number of suicides were reported, a priest in Louisiana among them. Speaking to the media after his death, the priest's wife said he'd found out his name was among those on the list before he killed himself. She said she would have forgiven her husband, and that God would have too. "God's grace in the midst of shame is the centre of the story for us, not the hack. My husband knew that grace, but somehow forgot that it was his when he took his own life."

During the early weeks of the crisis ALM, the company behind Ashley Madison, stopped responding in any sort of adequate way to calls and emails from its terrified customers. Countless marriages were at risk, people teetered on appalling decisions, and meanwhile ALM put out brisk press releases, one announcing the departure of CEO Noel Biderman. It made superficial adjustments to the front of its website, at some point deciding to remove the graphic that described Ashley Madison as "100% discreet".

Facebook Twitter Pinterest

'I was basically a therapist for them': Australian journalist Kristen Brown, who spoke to about 200 of those affected. Photograph: courtesy Kristen V Brown

So the masses sent spinning by the leak could not turn to ALM for advice. Most could not easily turn to their partners. Someone had to fill this enormous absence, hear grievances. Troy Hunt, a mild-mannered technology consultant from Sydney, had not expected it would be him.

As the crisis developed he found that dozens and then hundreds of people, caught up in the event, were looking to him for help and for counsel. Hunt, who is in his late 30s, explained what happened. His expertise is internet security; he teaches courses in it. As a side project, since 2013, he has run a free web service, HaveIBeenPwned.com, that allows concerned citizens of the internet to enter their email address, go through a simple process of verification, and then learn whether their personal information has ever been stolen or otherwise exposed in a data breach. When hackers pinched data from servers at Tesco, at Adobe, at Domino's Pizza, Hunt trawled through the data that leaked and updated his site so that people could quickly find out if they were affected. After the Ashley Madison leak he did the same.

Only this time, Hunt recalled, desperate and difficult and extremely personal messages began arriving in his inbox almost immediately. Mostly it was men who emailed - paying customers of Ashley Madison who mistakenly believed that Hunt, having sifted through the leaked data, might be able to help them. Could he somehow scrub their credit cards from the list? Hunt described the tone of these emails as fearful, illogical, "emotionally distraught". About a hundred emails a day arrived in that early period, Hunt recalls. Considered together they form a bleak and fascinating historical document: a clear view into the hivemind of those caught up in the leak, caught out.

People confessed to Hunt their reasons for subscribing to Ashley Madison in the first place: "I joined Ashley Madison one night bored, honestly... Curiosity... Drunken evening..." They volunteered to him what they'd done, or nearly done, or hadn't done at all. They described what it was like to learn about the leak: "The worst night of my life... Sheer fear... Sick and foolish... I can't sleep or eat, and on top of that I am trying to hide that something is wrong from my wife..." They pleaded with Hunt (who could do nothing for them). They apologised to him (a stranger). They wondered if they should admit everything to the people who mattered to them. And they wondered what that might cost. "Tell your wife and kids you love them tonight," said one email. "I shall do the same, as I really don't know if I will have many more chances to do so."

Some of those who got in touch, Hunt told me, mentioned suicide. He didn't know what to do. He was a computer consultant. He sent back the numbers of telephone helplines.

Who was behind the hack? Who was the Impact Team that claimed responsibility?

Troy Hunt often wondered about that. He knew a lot about data theft at big corporations, what it tended to look like. Hunt thought this episode seemed "out of character" with many such hacks he'd seen. The theft of such a large amount of data usually suggested to Hunt that somebody employed by the company (or someone who had physical access to its servers) was the culprit. But then, he reasoned, the subsequent leaks had been so careful, so deliberate. "They came out and said: 'This is what we're going to do.' Then radio silence. And then a month later: 'Here's all the data.'" It was sinister, Hunt thought, militaristic even.

Then there was the jarring strand of moralising in the messages the Impact Team did put out. "Learn your lesson and make amends" was the group's advice to any of Ashley Madison's users left in pieces by their work. Not the obvious behaviour, Hunt suggested, of a revenge-minded staffer who only wanted to hurt his or her employer.

Brian Krebs made efforts to understand the hackers, too. He'd never been able to figure out who first tipped him off, but he wondered at one point if he'd found a promising lead. In a detailed blog, published in late August, Krebs followed a trail of clues to a Twitter user who seemed to have suspicious early knowledge of the leak. "I wasn't saying they did it," Krebs told me, "I was just saying that maybe this was [a line of investigation] that deserved more attention." He didn't know if police forces investigating the case ever followed up on his lead. The Toronto force, to date, has announced no arrests. (When I asked, recently, if there had been any developments their press department did not reply.)

Krebs told me: "Whoever's responsible - no doubt they know that there are now lots of people wanting to put a bullet in their head. If it were me, if I was going to do something like this, I would make pretty darn sure that nobody could trace it back to me." At least in public, the Impact Team has not been heard from again.

What motivated the hackers, then? In the initial ransom note the Impact Team suggested that unseemly business practices at ALM - for instance a policy of charging users to delete their accounts on Ashley Madison and then continuing to store departing users' personal information on internal servers - had provoked the hackers' ire and justified its attack. But the mass release of private data, to make a point about the maltreatment of private data, cannot have seemed to anyone a very coherent reason for doing all this.

To try to better understand the thinking of the Impact Team I spoke to hackers who said they were not involved with the Ashley Madison attack but had kept a close eye on it. The general assumption, in this community, seemed to be that attacking a firm such as Avid Life Media (a bit shouty, a bit sleazy) was fair game. Few felt the mass release of millions of people's personal information - they called it "doxing" - was ideal hacker etiquette though. "Not sure I would have doxed 20 million people at the same time," one said. Even so they felt the saga would teach the world a useful lesson. "Anyone doing anything online," I was told, "should assume it isn't secure."

One hacker I spoke to said he'd spent hours and hours digging through the Ashley Madison data after the leak, going out of his way to draw attention to his most salacious findings. Speaking to me by email and in private chatrooms, he asked that I call him AMLolz, for "Ashley Madison laughs". We discussed some of the findings he'd made and subsequently publicised, through an AMLolz Twitter feed and an AMLolz website. He noted with some pride that in one of his deep searches he'd come across emails that suggested members of Ashley Madison's staff were themselves having extramarital affairs. He had posted screenshots of incriminating personal messages, and several magazines and newspapers had picked up on his findings and run stories.

AMLolz might not have been involved in the Ashley Madison hack, but he was certainly involved in giving it an impactful afterlife. I asked him what motivated him. Disapproval? Revenge? "Because it was very humorous," he said eventually. "And very interesting. No mission statement, just looking for lols."

AMLolz used the term "peripheral damage" more than once in conversation, neatly encompassing, in those words, all the sleepless unfaithful and their tortured other halves, the newly unemployed, the dead, their doubly grieving widows. I asked AMLolz what he would tell one of these "peripherally damaged" if he were to meet them in person.

He replied: "It would depend what they had to say to me first. [Smiley face.] That being said, something along the lines of: 'Own your actions. Don't lie to yourself, or anyone else...' It's not good. [Thoughtful face.]"

In the west of England, Michael could hardly disagree with this. Even as he sat in his home office, reading the developing news about Ashley Madison and wondering if his wife was doing the same, he was well aware of his own culpability. He didn't think he had anyone else to blame but himself. Who was he really going to blame? Ashley Madison? "I think it would probably be a little naive of me to expect high standards from a company that was promoting itself as a meeting point for people looking for adulterous affairs. It's a bit like borrowing money off your drug dealer and expecting him to pay it back." Michael simply accepted what was going on and watched, with a numb fascination, as the crisis rolled on.

In August, the private detective industry reported, cheerfully, an uptick in business. Lawyers steered high-publicity legal actions against Ashley Madison - at least three plaintiffs in America wanted to sue - as well as seeing through quieter divorce claims. In Australia a DJ decided to tell a woman live on air that her husband was on the database. Members and former members began to be sent anonymous extortion letters. Michael received several. Pay us in seven days, he was threatened in one email, "or you know what will happen... You can inform authorities but they can't help you. We are porfessionals [sic]." Michael was unnerved by the emails but ignored them. The world, in these small increments, got shabbier.

Like Troy Hunt in Australia, Kristen Brown, in California, found herself operating as a sort of on-the-go counsellor during these strange months. For Brown, a 29-year-old journalist, it began when she started interviewing victims of the Ashley Madison leak for the website Fusion.net. Interviewees kept wanting to talk, though, long after she'd published - a lot of these people, Brown guessed, left without anyone else they could speak to frankly. "I was basically functioning as a therapist for them. They were crushed by what happened." Brown guessed she'd spoken to about 200 of those affected by the hack over the past six months.

To an unusual degree, Brown thought, a tone of moral judgment skewed the commentary and discussion around the Ashley Madison affair. "It's a gut reaction, to pass a moral judgement," she said. "Because nobody likes the idea of being cheated on themselves. You don't want to find your own partner on Ashley Madison. But spending hours and hours on the phone with these people, it became so clear to me how frigging complicated relationships are."

Facebook Twitter Pinterest

'Maybe we need privacy disasters like this to help us wake up': Brian Krebs, the cybercrime journalist who broke the Ashley Madison story in July 2015. Photograph: Daniel Rosenbaum/New York Times/Redux/Eyevine

Brown continued: "We all have this idea of the site as completely salacious, right? Cheating men cheating on their unassuming wives. And I did speak to those men. But then I spoke to others who'd, say, been with their wife since they were 19 - they loved their wives but there were problems, there were kids, they'd stopped sleeping together. They had good partnerships, their lives worked, they didn't want to upend everything. They just weren't fulfilled or satisfied romantically. Some people were on the site with the permission of their spouses. I talked to one woman who was afraid to leave her husband, and being on Ashley Madison was her way of working out what to do. Some people I spoke to were single and didn't want attachment and using Ashley Madison was just a way. People's reasons were complex. They were real."

This, more or less, had always been Michael's reasoning for cheating. His situation was complex, and real. He told me he had been unfaithful to his wife "from after we first got married", conducting a string of one-off or months- or years-long affairs for almost 30 years. "As life partners, my wife and I fit really well. We are very, very good friends - that describes us. But I know there's a missing dimension to our relationship."

Ashley Madison was a way to try to account for that missing dimension.

And not always, said Michael, a particularly satisfying way. He wasn't even sure that every woman he spoke to during his time on the site was genuine. Sometimes, when conversation had a flavour of "classic soft porn", he said, he wondered if his correspondents were employees of the company, reading from scripts. (The likely truth, as suggested by internal documentation made available in the leak, was stranger still. Coders at Ashley Madison had created a network of fake, flirtatious chatbots to converse with men like Michael, teasing them into maintaining their subscriptions on the site. It was for this reason that commentators began to doubt whether Ashley Madison had as many subscribers as it advertised; Avid Life Media, ever since the leak, has always claimed to have a healthy and even growing userbase.)

Michael had met someone real through Ashley Madison. Like him she was in a stable companionable marriage, only one that lacked a certain dimension. She lived in the north of England. She had children. She and Michael shared tastes in books and spoke a lot on the phone. Sometimes they discussed their partners and their respective marriages, other times they steered from the subject. There was a sexual element to the affair, Michael said, but they never slept together. It was a relationship that was precious to him.

"If you're going to chat a woman up in a bar, or at a work conference, or wherever," Michael told me, "then: 'Hello, I'm married' is not a good opening line. Whereas if you're going on to a website like Ashley Madison - they know. It's a bit ridiculous to talk about honesty in terms of these relationships. But they actually start with honesty. Because you're not pretending to be something you're not."

Ashley Madison was a way of having a "safe affair", he said. Safe in the sense that he didn't think it likely he'd be found out by his wife (he had his special browser, his secret email address). Also safe in the sense that he didn't think anyone would get hurt.Since the leak Michael had not used Ashley Madison again nor spoken to the woman in the north. His wife, as of February 2016, had not found out about his affairs.

The hack of Ashley Madison was historic - the first leak of the online era to expose to mass view not passwords, not pictures, not diplomatic gossip, not military secrets, but something weirder, deeper, less tangible. This was a leak of desires.

"I think that history's probably littered with examples of madams whose little black book went walking, you know what I mean?" said Brian Krebs. "But this was massive, en masse, on the internet. Who knows? Maybe we need privacy disasters like this to help us wake up."

Kristen Brown thought it was important to take away a different instruction from the saga. That marriage is not one thing, and that the millions of users of Ashley Madison very likely had millions of different reasons for being on there. "There's a vibe between two people that can't be quantified. How to say what the right path is for any one pair? Relationships are fucking weird. And they get weirder the longer they go on."

In London recently I met with Troy Hunt. He'd flown in from Australia to teach a corporate course on internet security. We had lunch between morning and afternoon sessions in his classroom in Canary Wharf. While we ate Hunt showed me his phone - another email had just come in from someone requesting his help. Six months had gone by since the leak; the flow of desperate messages had slowed but not stopped.

Hunt responded to this email the way he always did now, sending back a prewritten response that included a list of answers to frequently asked questions about the hack. Also that list of hotline numbers.

When we'd finished eating his teaching resumed. Two dozen people filed into the room with their laptops and sat quietly while Hunt lectured them about cyber security. He'd worked a contemporary lesson into his speech, and projecting an image of a now-infamous website on to a screen behind him, he said to the class: "Put up your hand if any of you have an account with Ashley Madison."

A nervous laugh went around the room. Nobody put up their hand.
Topics

# data[].attributes.attachment_link.description
It was 9 o'clock on a Sunday night last July when a journalist called Brian Krebs came upon the scoop of his life. The 42-year-old was at home in Virginia at the time, and wearing pyjamas. For years Krebs had written a popular blog about internet security, analysing thefts of consumer data from big companies around the world, Tesco, Adobe, Domino's Pizza among them.

# data[].attributes.attachment_link.readable_text
What: Stop kidding yourself that you only re-use passwords on accounts that don't matter, or that you have an unbreakable password scheme that no one else can guess. Every single thing with a password needs to have a unique password, shared with nothing else.

What: Stop kidding yourself that you only re-use passwords on accounts that don't matter, or that you have an unbreakable password scheme that no one else can guess. Every single thing with a password needs to have a unique password, shared with nothing else.

Why: Services get hacked, with entire databases of passwords published in the open. People get "phished", tricked into entering their passwords into shady imitations of the sites they intended to visit. If this happens, you want to limit the damage, ensuring that only one site gets breached.

How: Unless you absolutely categorically have a reason not to ...

2. Use a password manager

What: Software like LastPass (free) or 1Password ($2.99/month or $49), which will store your passwords, generate secure random ones for you, and sync them across multiple devices.

Why: If you can memorise all your passwords, you can almost guarantee that they aren't varied enough to be secure. A password manager may feel like putting all your eggs in one basket, but it's a padded secure basket kept up-to-date by the best minds in the basket business, and what you're doing right now is more like juggling the eggs above your head while blindfolded.

How: Download the password manager, install it on your desktop (you can do mobile later), and start running it. You don't even have to change your passwords all at once: the manager will notice when you log in, and ask you whether you want to save the new password. That should be your cue to create a new one.

3) Use random passwords

What: Once you've got your password manager, use it to generate secure random passwords for you, rather than trying to invent your own.

Why: You aren't as random as you think, and "brute forcing" passwords - systematically trying every variation until you succeed - is getting quicker at the same rate computers are. If you have a handy method for creating passwords, like "take the first letter of every word in a line of poetry", then someone else has probably also realised the same thing, and written a programme to automatically guess those passwords. Try searching Google for " tbontbtitq " (or "to be or not to be, that is the question") if you don't believe me.

How: You've already got your password manager set up, yes? Even if you haven't, some browsers will do it for you. Apple's Safari, for instance, will happily generate random passwords when signing up for new accounts, then store them in iCloud Keychain.

4. Turn on two-step verification everywhere you can

What: Many services, including Facebook, Google, Twitter, Tumblr and more, let you enable two-step verification, also known as two-factor authentication. As well as a password, you need to prove you have access to a second trusted device, normally a phone, to log on. How you prove that varies: sometimes a text is sent, sometimes you use a special app, sometimes you just hit a notification on your phone.

Facebook Twitter Pinterest

2-step-verification-code Photograph: Samuel Gibbs for the Guardian

Why: Two-step verification prevents a third-party from logging in to your accounts even if they have managed to steal your password. It's an added layer of security, which makes it very difficult indeed to hack in to protect accounts.

How: Every service has a different method for enabling the process, which hurts take-up, but handy resource Turn On 2FA will walk you through it for all the sites you use.

5. Update your software

What: Most software has an automatic update function. Use it.

Why: Most hacks are carried out by attacking software using weaknesses that were known, and fixed, long ago. It's like we've invented vaccines, but you're still catching smallpox. Particular focus should be paid to your operating system, web browser, and Adobe Flash.

How: Enable automatic updates.

6. Put a six-digit PIN on your phone and set it to wipe if it's guessed wrongly too many times

Facebook Twitter Pinterest

On an iPhone, open settings, hit Touch ID & Passcode, flick on Erase Data, and click Change Passcode to set it to a six-digit PIN. Photograph: Bloomberg/Bloomberg via Getty Images

What: Your phone has the ability to require a PIN before it is unlocked it. Use it.

Why: If your phone gets taken while it's unlocked, there's not much you can do. But if it's locked when it gets stolen, you can prevent the bad loss of hundreds of pounds of technology from turning into the loss of enough personal data to have your identity stolen too.

How: On an iPhone, open settings, hit Touch ID & Passcode, flick on Erase Data, and click Change Passcode to set it to a six-digit PIN. Almost every Android is different, but look for a "security" menu in the settings app, sometimes under "personal". Then, head to the "lock screen" menu to enable the auto-erase feature.

7. Enable full-disk encryption

What: Your computer's hard drive can be set to automatically encrypt when it's turned off.

Why: You think the risk of identity theft is bad when your phone is stolen, just think what happens when your computer is lifted.

How: On a Mac, enable FileVault; on Windows, turn on BitLocker.

8. Back-up to an external hard drive

What: Everything on your computer should be stored on a physically separate hard drive under your possession. Ideally, everything on your phone should be stored on your computer which should then be etc etc.

Why: If the worst happens, and you lose everything, you need to be able to restore. This could happen because of a ransomware attack, because someone decided to personally ruin your life, or just because of a literal lighting strike. Cloud storage will help, but cloud platforms go bust unexpectedly, are just as vulnerable to hacking, and have an annoying tendency to "mirror" your computer - meaning something deleted from your local storage can be deleted from the cloud at the same time.

How: Buy a cheap USB hard drive. If you use a Mac, just leave it plugged in and enable Time Machine; if you have a Windows PC, plug it in and follow these instructions.

Yahoo hack: 1bn accounts compromised by biggest data breach in history
Topics

# data[].attributes.attachment_link.title
Eight things you need to do right now to protect yourself online

# data[].attributes.attachment_link.description
What: Stop kidding yourself that you only re-use passwords on accounts that don't matter, or that you have an unbreakable password scheme that no one else can guess. Every single thing with a password needs to have a unique password, shared with nothing else.

# data[].attributes.attachment_link.readable_text
The Investigatory Powers Bill - nicknamed the Snoopers' Charter - was agreed upon by both Houses of Parliament and passed into law by Royal Assent on 29 November 2016, making it the Investigatory Powers Act.

However, concerns around the balance between privacy and security have dogged the bill since its inception, and some of its more controversial elements will have to be changed following a successful legal challenge in January 2018.

A UK Court of Appeal ruling found the Data Retention and Investigatory Powers Act (DRIPA) - a previous law covering state surveillance which has been expanded upon with the Investigatory Powers Act of 2016 - as unlawful.

The court ruled that the legislation breached British people's rights by collecting internet activity and phone records and letting public bodies grant themselves access to these personal details with no suspicion of serious crime and no independent sign-off. The latest legislation will have to be "urgently changed" as a result.
Read next: Snoopers Charter ruled unlawful by UK Court of Appeal
The challenge was brought by Labour deputy leader Tom Watson MP, represented by civil liberties charity Liberty, which is challenging this latest law in a separate case in the High Court to be heard later this year.

Martha Spurrier, Liberty's director, said: "Yet again a UK court has ruled the government's extreme mass surveillance regime unlawful. This judgment tells ministers in crystal clear terms that they are breaching the public's human rights. The latest incarnation of the Snoopers' Charter, the Investigatory Powers Act, must be changed."

Fresh amendments were proposed by the government in November 2017 following a European court ruling which said the "general and indiscriminate retention" of personal communications data by police and security services "cannot be considered justified within a democratic society".

The proposed provisions include:

Requests for personal communications data by the police and other public bodies to be restricted to investigations into crimes that carry a prison sentence of at least six months.

The introduction of independent authorisation of communications data requests by a new body, known as the Office for Communications Data Authorisations, under the investigatory powers commissioner, Lord Justice Fulford.

Restricting the use of communications data to investigations into serious crime.

Additional safeguards which must be taken into account before a data retention notice can be given to a telecommunications or postal operator.

Clarification of the circumstances in which notification of those whose communications data has been accessed can occur.

Mandatory guidance on the protection of retained data in line with European data protection standards.

Still, the main powers of the law as it stands are:
Web and phone companies (CSPs) will store records of websites visited by every customer for 12 months for access by police, security services and other public bodies upon issue of a warrant.

In practice this would take the form of an itemised list of each citizen ' s browsing history. This would not be a list of the specific web pages but the main domain (so computerworlduk.com but not the specific stories you read) so a basic online footprint can be drawn up. One concern here will be around the security of this data, especially in the current climate of TalkTalk customer hacks and data dumps.

However, they won't be able to track sites accessed over various . VPNs See: The best VPNs list

- Security services will be legally empowered to bug computers and phones upon approval of a warrant. Companies will be legally obliged to assist these operations and bypass encryption where possible (more on this below).

- Security services can acquire and analyse bulk collections of communications data. For example, this could mean a bulk data set such as NHS health records.

- Oversight for these operations will come with a new "double-lock", where any intercept warrants will need ministerial authorisation before being adjudicated by a panel of judges, who will be given power of veto. This panel will be overseen by a single senior judge, the newly created Investigatory Powers Commissioner.

In May 2017, a leaked draft statutory instruments document detailed how the government is seeking to compel telecommunications operators to provide real-time access to named individuals' communications within one working day under the recently passed law. This includes encrypted messages.

The government also asks for the capability to "provide and maintain the capability to simultaneously intercept, or obtain secondary data" from 6,500 people at any one time. For more information visit our sister site Techworld

For some context, figures from the Home Office, as published by The Guardian , show there were 517,236 authorisations in 2014 of requests for communications data from the police and other public bodies and a further 2,765 interception warrants authorised by ministers.

Encryption concerns

Concerns over the British state's approach towards encryption dogged the government during the bill's passage through Parliament.

The issue for the security services would be that over the top communications providers - like Apple's iMessage and the popular WhatsApp messaging service - apply end-to-end encryption to all messages, meaning they can't read them even if they wanted, or were asked to.

In Apple's formal submission to the Bill Commission, the company voiced concern that: "Passages in the bill could give the government the power to demand Apple alters the way its messaging service, iMessage, works" in a way that gives security services the power to eavesdrop on messages, according to The Guardian. Apple CEO Tim Cook has been consistently outspoken in his defence of encryption.

Emails sent using Microsoft Outlook aren't automatically encrypted in this way and Gmail requires an end-to-end encryption extension for Chrome. Blackberry offers end-to-end encryption between devices through its paid BBM Protected product. The Cisco Spark messaging service has built in end-to-end encryption.

Earl Howe, minister of state for defence and deputy leader in the House of Lords, said following the second reading: "It may be entirely sensible for the government to work with [communication service providers] to determine whether it would be reasonably practicable to take steps to develop and maintain a technical capability to remove encryption that has been applied to communications or data.

"Law enforcement and the intelligence agencies must retain the ability to require telecommunications operators to remove encryption in limited circumstances," he said.
Read next: The Snooper's Charter still has an encryption problem
However, Nic Scott, managing director UK and Ireland at enterprise data security specialists Code42, makes the important observation that there are no "half measures" when it comes to encryption. He says: "You either have encryption in place or you don't. Once you create a backdoor for law enforcement purposes, you are also opening the door to other, potentially malicious, parties."

Significant Demands

The shadow home secretary Andy Burnham laid out six areas of concern for the Investigatory Powers Bill back in March 2016.

Speaking at the second reading of the bill, Burnham said: "This bill isn't yet good enough. Simply to block this legislation would in my view be irresponsible, it would leave police and security services in limbo. We must give them the tools to do their job. The public interest lies in getting this right and in not sacrificing quality to meet the deadline."

I want to see major changes to #IPBill. Will set out plan later to get them. My judgment is this will achieve more than outright opposition.
- Andy Burnham (@andyburnhammp) March 15, 2016

The six concerns were as follows:

1. Privacy: "The Home Secretary said [privacy] was hardwired into the bill, but I see them as more cosmetic changes and haven't directly answered the concerns of the joint committee." Burnham asked that the bill takes a "presumption of privacy".

2. Specific powers: "Internet connection records (ICRs) have been described as the modern equivalent of an itemised phone bill, however the joint committee noted that this is not a helpful description." Burnham went on to explain that there should be a "higher hurdle" for use of this power limited to cases of serious criminal activity rather than any crime. He also asked that the terms "national security" and "economic wellbeing" are defined more explicitly.

3. Internet Connection Records: "Definitions of ICRs (Clause 54) remain vague and I see them becoming more intrusive as technology advances. A stricter definition of what can be included in an ICR should be included. The current confusion is clouding this bill and needs to be clarified."

4. Bulk Powers: "Routine gathering of large quantities of information from ordinary people does lead to privacy concerns and should be as targeted as possible [...] It is for the government still to convince the public that these powers are needed." Burnham asked specifically for an independent review to conclude in time for report and third reading on this issue.

5. Judicial oversight: "The government has given significant ground on this issue and the bill is stronger as a result, however we believe it could be stronger still [...] I have previously shared concern that this leads to a narrower test looking at only the process and reasonableness of the home secretary's decision, rather than actual merits and substance of the warrant."

May had earlier reassured the house that judicial commissioners will have access to the same information about a warrant as the home secretary. Burnham recognised this, but continued: "If this is the case why not delete the judicial review clause? To make it absolutely clear that this is not just a double lock, but an equal lock."

6. Misuse of the powers: "There needs to be safeguards for the collection of data in a lawful manner and we must also agree that there needs to be an overarching law for the obtaining of data and any use that data is subsequently put to. Both should be a criminal offence."

First reading

The Home Office published its draft investigatory powers bill on November 4, 2015.

Following criticism from three joint committees: the science and technology committee on February 1, the intelligence and security committee on February 9 and most importantly, the joint committee for the bill itself on February 11, Home Secretary Theresa May revised the bill, claiming that it "reflects the majority of the committees' recommendations".

The joint committee for the draft investigatory powers bill made 86 recommendations for changes to the bill in its report, concentrating on issues of clarity, judicial oversight and justification of the various powers.

Addressing these specific concerns, May said of the revised bill: "We have strengthened safeguards, enhanced privacy protections and bolstered oversight arrangements."

Not everyone agreed with this assessment though. Dr. Gus Hosein, executive director of Privacy International said: "It would be shameful to even consider this change cosmetic [...] The continued inclusion of powers for bulk interception and bulk equipment interference - hacking by any other name - leaves the right to privacy dangerously undermined and the security of our infrastructure at risk."

The joint committee report

The joint committee for the bill issued its report, along with a list of suggested amendments for the bill, on February 11, 2016. The suggestions include:

- Clarification over the concept of end-to-end encryption: "The Government still needs to make explicit on the face of the bill that Communications Service Providers (CSPs) offering end-to-end encrypted communication or other un-decryptable communication services will not be expected to provide decrypted copies of those communications if it is not practicable for them to do so."

On the issue of encryption the government said the revised bill: "Clarifies the government's position on encryption, making it clear that companies can only be asked to remove encryption that they themselves have applied, and only where it is practicable for them to do so."

- CSPs being forced to retain internet history data of users should be provided with "whatever technical and financial support is necessary to safeguard the security of the retained data" but the government shouldn't be responsible for 100 percent of the costs.

The Don't Spy On Us coalition, which includes the Open Rights Group and Privacy International, responded: "Proposals to collect the internet connection records (ICRs) of every UK citizen could cost more than £1 billion." This figure is based on a similar scheme which has since been dropped in Denmark due to the cost. The initial Home Office estimate for the storage of these records was just £174m over ten years.

- "Fuller justification" for bulk surveillance: "We believe that that the lack of a formal case for bulk personal datasets (BPDs) remains a shortcoming when considering the appropriateness of this power."

- There should be no power to ask foreign intelligence agencies to undertake surveillance where the UK authorities cannot, for example in the USA.

The revised bill "explicitly bans our agencies from asking foreign intelligence agencies to undertake activity on their behalf unless they have a warrant approved by a Secretary of State and Judicial Commissioner."

- Hacking should be targeted: "Targeted interception and targeted equipment interference warrants cannot be used as a way to issue thematic warrants concerning a very large number of people."

- An annual report that must contain: "Information about the impact, results and extent of the use of powers in the bill so effective public and parliamentary scrutiny of the results of the powers can take place."

Writing for Wired.com, Liberal Democrat MP Lord Strasburger, who sat on the joint committee, said: "This bill is a long way from the finished article. It needs more than mere tweaking, it needs to be fundamentally rethought and rebuilt. The Home Office should stop rushing to push it through and take its time to get it right."

The intelligence and security committee criticised the bill for a lack of clarity and transparency around powers and suggested wide ranging amendments to the bill. Mostly though the committee says that the bill could benefit from starting again, saying: "The draft bill adopts a rather piecemeal approach, which lacks clarity and undermines the importance of the safeguards associated with these powers.

"We have therefore recommended that the new legislation contains an entirely new part dedicated to overarching privacy protections, which should form the backbone of the draft legislation around which the exceptional powers are then built. This will ensure that privacy is an integral part of the legislation rather than an add-on."

The science and technology select committee report confronted concerns over the impact the legislation could have on the UK's technology sector, equipment interference powers and a lack of clarity when it comes to the issue of encryption.

Nicola Blackwood MP, chair of the science and technology committee said: "It is vital we get the balance right between protecting our security and the health of our economy. We need our security services to be able to do their job and prevent terrorism, but as legislators we need to be careful not to inadvertently disadvantage the UK's rapidly growing tech sector."

The joint committee on the Draft Investigatory Powers Bill was sent 148 sets of evidence raising concerns and views about aspects of the legislation, and May faced questions over these concerns in front of the committee in December.

The politics

There was generally cross-party approval of the bill as first proposed, with Shadow Home Secretary Andy Burnham stating that it was "neither a snooper's charter nor a plan for mass surveillance."

Conservative MP David Davis has been one of the more outspoken critics of the proposed legislation. Talking to The Guardian he said: "In every other country in the world, post-Snowden, people are holding their government's feet to the fire on these issues, but in Britain we idly let this happen [...] Because for the past 200 years we haven't had a Stasi or a Gestapo, we are intellectually lazy about it, so it's an uphill battle."

Author and journalist Heather Brooke went one step further. Writing for The Guardian she said: "The spies have gone further than [George Orwell] could have imagined, creating in secret and without democratic authorisation the ultimate panopticon. Now they hope the British public will make it legitimate."

Edward Snowden tweeted: "By my read, #SnoopersCharter [The Draft Investigatory Powers Bill] legitimises mass surveillance. It is the most intrusive and least accountable surveillance regime in the West."

Apple submitted a formal submission to the bill committee, specifically around the issue of encryption, on Monday 21 December, expressing: "We believe it would be wrong to weaken security for hundreds of millions of law-abiding customers so that it will also be weaker for the very few who pose a threat. In this rapidly evolving cyber-threat environment, companies should remain free to implement strong encryption to protect customers," as per The Guardian.

According to the UK public generally approve of surveillance, with 44 percent of respondents stating it wouldn't bother them to know that they could be spied upon and they don't think they are at this time.

Obligations on communications service providers

The use of investigatory powers relies heavily on the cooperation of so-called 'communications service providers' (CSPs) in the UK and overseas. The draft bill clearly outlined a legal duty on British companies to assist in hacking devices (equipment interference warrants).

On the issue of data retention May told the joint committee in January that: "There have been discussions with providers. CSPs have shown me responsiveness on that matter." However, she avoided giving any detail on how this would work in practice and how much it might cost, saying: "There are no exact figures, I'm happy to provide written evidence of Home Office work in this area."

The science and technology joint committee report pushed the government for greater clarification when it comes to practical concerns around the retention of this data. The report reads: "There seems still to be confusion about the extent to which 'internet connection records' will have to be collected. This in turn is causing concerns about what the new measures will mean for business plans, costs and competitiveness."

The joint committee has pushed the government to provide: "Whatever technical and financial support is necessary to safeguard the security of the retained data" in its report, but suggests that the government shouldn't be responsible for 100 percent of the costs.

A spokesperson for UK internet service provider (ISP) BT responded to this obligation by stating: "National security is a critical issue and everyone needs to play their part, including industry. Parliament has long taken the view that the national interest is best served by allowing security and law enforcement authorities access to certain types of data under certain circumstances. We believe there must be a clear legal framework around this regime, one that ensures adequate checks and balances are in place to weigh up any human rights concerns."

ISP Virgin Media said it "does not monitor or control what customers do online but complies with all lawful requests. It is for Parliament to decide where the balance lies between the needs of law enforcement and citizens' privacy.''

ISPs have been cooperating with requests like this since 1984 under obligations outlined in the Telecommunications Act, if requested by the Secretary of State in the interest of national security. This bill looks to write this power into law for the security and intelligence agencies.

The draft bill also outlined a means for ISPs, telecommunications operators and postal operators to receive appropriate contributions to cover the additional costs of these activities.

These providers can appeal requests for data, but only directly to the Secretary of State.

# data[].attributes.attachment_link.title
Investigatory Powers Act: What you need to know

# data[].attributes.attachment_link.description
The Investigatory Powers Bill - nicknamed the Snoopers' Charter - was agreed upon by both Houses of Parliament and passed into law by Royal Assent on 29 November 2016, making it the Investigatory Powers Act.

# data[].attributes.attachment_link.readable_text
A Canadian journalist has shared the story of how she fought back against a 15-year-old who hurled insults at her online, confronting his family and school over his behaviour.

When the online insults started to roll in thick and fast, telling Jessica Davey-Quantick she was disgusting, taking aim at her weight and telling her she should kill herself, she decided to take action.

The comments began appearing last week beneath an Instagram photograph of Davey-Quantick performing at a burlesque show in January. She responded swiftly, hoping to shame the commenter into retreating and perhaps apologising.

Instead she awoke to find a stream of hurtful remarks left on several more of her posts, telling her to "slit my own wrists and eat my own blubber".

When, with the help of some friends, she discovered the commenter was a 15-year-old from the UK she decided to confront his father and headteacher in London. "I am over the idea that we have to somehow protect the individuals [involved]," Davey-Quantick told the Canadian Broadcasting Corporation.

"What happens when he tells a 15-year-old girl to kill herself?" she said. "What happens when he tells somebody that they're disgusting? Is that going to lead to somebody in the bathroom putting their fingers down their throat? Is that going to lead to someone committing suicide? It's dangerous."

With the help of friends, she tracked down the abuser's Facebook page and contacted his family and school.

Initially the boy's father apologised, but later claimed his son had not been involved, she said. He then told her that she should be ashamed for targeting his son before suggesting that she keep her Instagram posts private to avoid abuse.

This advice is routinely doled out in response to internet bullying, she said. "And for me that's troubling. Because what we're doing then is we're ceding public spaces to trolls. We're giving them the power."

The teenager's school, appalled at the allegations, asked Davey-Quantick to send screenshots of the comments.

The next message she received was from the teenager himself. He and his friends had been suspended from school and had had their phones taken away.

"More importantly, he was so apologetic," Davey-Quantick said. "And I got this email from him basically telling me he is so ashamed of himself. He is so ashamed that he has done this to his parents, to his friends, that he doesn't do this normally, and that he's learned his lesson."

She said the email had reinforced her belief in the need to continue calling out those who use the anonymity of the online world to abuse others.

"These people who do it generally have private pages," said Davey-Quantick. "So we need to take it to their world.

"I'm probably going to keep getting [abusive messages], and I'm probably going to keep posting and I'm probably going to keep finding their mothers."
Topics

# data[].attributes.attachment_link.title
Canadian shames teenage troll in UK over burlesque photo insults

# data[].attributes.attachment_link.description
A Canadian journalist has shared the story of how she fought back against a 15-year-old who hurled insults at her online, confronting his family and school over his behaviour. When the online insults started to roll in thick and fast, telling Jessica Davey-Quantick she was disgusting, taking aim at her weight and telling her she should kill herself, she decided to take action.

# data[].attributes.attachment_link.readable_text
Privacy International, a UK privacy group, and the U.S.-based Electronic Privacy Information Center have put together a world map of surveillance societies, rating various nations for their civil liberties records.

Both the U.S. and the UK are colored black for "endemic surveillance," as are Thailand, Taiwan, Singapore, Russia, China and Malaysia.

Among the trends that the two organizations have tracked:

``

* The 2007 rankings indicate an overall worsening of privacy protection across the world, reflecting an increase in surveillance and a declining performance on privacy safeguards.

* Concern over immigration and border control dominated the world agenda in 2007. Countries have moved swiftly to implement database, identity and fingerprinting systems, often without regard to the privacy implications for their own citizens

* The 2007 rankings show an increasing trend amongst governments to archive data on the geographic, communications and financial records of all their citizens and residents. This trend leads to the conclusion that all citizens, regardless of legal status, are under suspicion.

* The privacy trends have been fueled by the emergence of a profitable surveillance industry dominated by global IT companies and the creation of numerous international treaties that frequently operate outside judicial or democratic processes.

* Despite political shifts in the US Congress, surveillance initiatives in the US continue to expand, affecting visitors and citizens alike.

* Surveillance initiatives initiated by Brussels have caused a substantial decline in privacy across Europe, eroding protections even in those countries that have shown a traditionally high regard for privacy.

* The privacy performance of older democracies in Europe is generally failing, while the performance of newer democracies is becoming generally stronger.

* The lowest ranking countries in the survey continue to be Malaysia, Russia and China. The highest-ranking countries in 2007 are Greece, Romania and Canada.

* The 2006 leader, Germany, slipped significantly in the 2007 rankings, dropping from 1st to 7th place behind Portugal and Slovenia.

* In terms of statutory protections and privacy enforcement, the US is the worst ranking country in the democratic world. In terms of overall privacy protection the United States has performed very poorly, being out-ranked by both India and the Philippines and falling into the "black" category, denoting endemic surveillance.

* The worst ranking EU country is the United Kingdom, which again fell into the "black" category along with Russia and Singapore. However for the first time Scotland has been given its own ranking score and performed significantly better than England & Wales.

* Argentina scored higher than 18 of the 27 EU countries.

* Australia ranks higher than Slovakia but lower than South Africa and New Zealand.

UPDATE: As requested, here's a link to the data. Apologies for not including it in the original post.

Image: Privacy International and EPIC

# data[].attributes.attachment_link.title
World's Top Surveillance Societies -- Updated with link

# data[].attributes.attachment_link.description
Privacy International, a UK privacy group, and the U.S.-based Electronic Privacy Information Center have put together a world map of surveillance societies, rating various nations for their civil liberties records. Both the U.S. and the UK are colored black for "endemic surveillance," as are Thailand, Taiwan, Singapore, Russia, China and Malaysia.

# data[].attributes.attachment_link.readable_text
Governments around the world are tightening control over citizens' data and using claims of "fake news" to suppress dissent, eroding trust in the internet as well as the foundations of democracy, according to Freedom on the Net 2018.

At the same time, the regime in China has become more brazen in providing like-minded governments with technology and training that enable them to control their own citizens.

"Democracies are struggling in the digital age, while China is exporting its model of censorship and surveillance to control information both inside and outside its borders," said Michael J. Abramowitz, president of Freedom House.

These trends led global internet freedom to decline for the eighth consecutive year in 2018.

"This year has proved that the internet can be used to disrupt democracies as surely as it can destabilize dictatorships," said Adrian Shahbaz, Freedom House's research director for technology and democracy. "Online propaganda and disinformation have increasingly poisoned the digital sphere, while the unbridled collection of personal data is breaking down traditional notions of privacy."
Key findings from Freedom on the Net 2018:

Declines outnumber gains for the eighth consecutive year. Out of the 65 countries assessed in Freedom on the Net, 26 experienced a deterioration in internet freedom. Almost half of all declines were related to elections.

trains the world in digital authoritarianism: Chinese officials held trainings and seminars on new media or information management with representatives from 36 out of the 65 countries assessed by Freedom on the Net.

Internet freedom declined in the United States.

Citing fake news, governments curbed online dissent: At least 17 countries approved or proposed laws that would restrict online media in the name of fighting "fake news" and online manipulation.

Authorities demand control over personal data: Governments in 18 countries increased surveillance, often eschewing independent oversight and weakening encryption in order to gain unfettered access to data.

Major Developments

# data[].attributes.attachment_link.description
Governments around the world are tightening control over citizens' data and using claims of "fake news" to suppress dissent, eroding trust in the internet as well as the foundations of democracy, according to Freedom on the Net 2018.