# data[].attributes.attachment_link.readable_text
An Exercise in Language Compression

Are Pop Lyrics Getting More Repetitive?

In 1977, the great computer scientist Donald Knuth published a paper called The Complexity of Songs, which is basically one long joke about the repetitive lyrics of newfangled music (example quote: "the advent of modern drugs has led to demands for still less memory, and the ultimate improvement of Theorem 1 has consequently just been announced").

I'm going to try to test this hypothesis with data. I'll be analyzing the repetitiveness of a dataset of 15,000 songs that charted on the Billboard Hot 100 between 1958 and 2017.

Measuring repetitiveness?

I know a repetitive song when I hear one, but translating that intuition into a number isn't easy. One thing we might try is looking at the number of unique words in a song, as a fraction of the total number of words. But this metric would call the following lyric excerpts equally repetitive:

baby I don't need dollar bills to have fun tonight
I love cheap thrills
baby I don't need dollar bills to have fun tonight
I love cheap thrills
I don't need no money
as long as I can feel the beat
I don't need no money
as long as I keep dancing
~ Sia, Cheap Thrills

tonight I need dollar bills
I don't keep fun
cheap thrills long to feel money
the bills don't need the dancing baby
fun dollar dancing thrills the baby I need
don't have fun
no no don't have dancing fun tonight
beat the can as I don't feel thrills
love the dancing money
~ Colin Morris, Original composition

These are both 52 words long and use the same 23 word vocabulary, but the first one is clearly more repetitive, because it arranges words in a predictable, repetitive order.

Repetitiveness ≈ compressibility?

You may not have heard of the Lempel-Ziv algorithm, but you probably use it every day. It's a lossless compression algorithm that powers gifs, pngs, and most archive formats (zip, gzip, rar...).

What does this have to do with pop music? The Lempel-Ziv algorithm works by exploiting repeated sequences. How efficiently LZ can compress a text is directly related to the number and length of the repeated sections in that text.

Here's how it works.

You can explore some examples of the algorithm applied to full songs here. It turns out, for example, that the entire lyrics of Cheap Thrills reduce in size 76% when compressed.

Is 76% a lot? Here's the distribution of size reduction percentages across (almost) all 15,000 songs in my dataset:

The Repetition of Pop Music

Distribution of compressibility of 15,000 songs from 1958 to 2017, excluding 20 outliers.

You may have noticed the percentages on the x-axis are not evenly spaced. I'm using a logarithmic scale with the property that, for any given song, a song that compresses half as small is a fixed distance away. For example, the distance between 20% and 60% is the same as between 98% and 99%. I'll be using this kind of scale throughout.

I excluded the 20 most repetitive songs in the dataset from the above chart. When you see what it looks like with them, you'll understand why:

What are these uber-repetitive outliers? Around The World by Daft Punk gets reduced a whopping 98%. It goes from 2,610 characters to 61. Small enough to fit in a tweet - twice! ( Here are the lyrics, if you're not familiar with the song.) Check out some of the others below.

The Most Repetitive Songs

Of 15,000 songs from the Billboard Hot 100

Intuitively, Around The World definitely feels like a very repetitive song. Several of the familiar entries on this list are essentially novelty songs (Macarena, Barbara Streisand, The Thong Song, Cotton Eye Joe...) famed for their silly refrains. This inspires some confidence in the metric I'm using.

So, is music getting more repetitive? The current decade is pretty well-represented in the top 10 above, but it's also a bit overrepresented in my dataset (it's easier to find lyrics for recent songs). We'll need to look at a lot more data to be sure...

Who's responsible for this madness?

Let's look at the average repetitiveness of some prolific artists in the dataset (those that have at least 15 charting songs as solo artists).

Repetitiveness Per Artist

Genre does seem like a differentiating factor here. In the 00's, our artists actually separate pretty cleanly into two clusters, with country music and hip-hop (and whatever John Mayer does) on the left, and pop and rock on the right.

The variation between artists is considerable. The Backstreet Boys have an average compression ratio of 60%, to Brad Paisley's 38%. In other words, if we asked the Backstreet Boys and Brad Paisley to each write a 400 word song, and compressed them both, we'd expect Brad's compressed song to be 50% bigger than BSB's.

Let's zoom in on a specific artist, say Gwen Stefani. Each circle below represents a song in her discography. The background blob is the histogram of all songs in the dataset (the same one as before, but mirrored).

On average, Gwen's music is extremely repetitive, but there's a wide spread between the incessantly repetitive "Hollaback Girl", and the laid-back ballad "Cool", which is far less repetitive than the median song in the dataset.

You can search for your favorite artist in the dropdown above. A few interesting examples (click an artist's name to jump to their chart):

People have described 1989 as the album where Taylor Swift fully transitioned from country singer/songwriter to popstar. This data adds some flavour to that claim. 4 of her 5 most repetitive songs are from 1989.

Madonna has a long, consistent legacy of highly repetitive music. Interestingly, her two least repetitive songs are arguably the only two that don't belong to the pop genre: they're both from the musical Evita, and written by Tim Rice and Andrew Lloyd Weber.

Rappers like J. Cole and Eminem tend to be consistently non-repetitive.

Is it any wonder Rihanna won the title of most repetitive artist in the dataset? Over half a dozen of her songs sit on the far right of the distribution (including, appropriately, Pon de Replay).

# data[].attributes.attachment_link.description
Measuring repetition in song lyrics over time using a compression algorithm

# data[].attributes.attachment_link.title
Police killing rates in G7 members [OC] * r/dataisbeautiful

# data[].attributes.attachment_link.description
41337 points and 10040 comments so far on reddit

# data[].attributes.body
Hoe maak je een goede datavisualisatie.

# data[].attributes.attachment_link.readable_text
FOSE

The mosaic effect and big data

The proliferation of government data sets is providing developers with ample fodder for writing useful and potentially profitable applications around census, weather, health, energy, business, agricultural and other information. But as the government makes more and more data discoverable and machine readable, there is the threat that disparate threads can be pieced together in a way that yields information that is supposed to be private.

This kind of analysis through the combination of big data sets is called the mosaic effect. And it isn't necessarily bad, Marion Royal, director of Data.gov at the General Services Administration, said at a May 13 FOSE session. He noted, for example, that the combination of big data sets can supply clues on the paths of seasonal flu outbreaks. But there is also the potential for a bad guy to, say, use transportation data and energy production data to figure out where oil and gas are moving on trains and trucks.

The White House publicly released its Open Data Action Plan on May 9, the one-year anniversary of President Barack Obama's executive order that made open data the default setting of the federal government. According to Royal, the government has found "very few instances of agencies putting up data with sensitivities."

The action plan aggregates planned release schedules for agency data sets, including information on health, climate, small business and manufacturing opportunities, crime, education, and public domain information on the federal workforce.

While the government is taking steps to reduce the exposure of personally identifiable information or security threats, the lingering problem is that it is impossible to scope out all the potential future uses of government datasets in advance, said David E. McClure, who works on open data at the National Oceanic and Atmospheric Administration.

"We know there's undiscovered value and unrecognized threats," McClure said. "We need to have some way to manage it and the short answer is, I don't know how to."

Royal suggested that the model of preserving privacy by individual consent might be obsolete when so much data is passively captured by sensors, and the abundance of social media and search data collected by private companies makes anonymization "virtually impossible," he said: "Privacy as a concept is becoming less clear as technology increases and big data becomes more prevalent, and available."

About the Author

Adam Mazmanian is executive editor of FCW.

Before joining the editing team, Mazmanian was an FCW staff writer covering Congress, government-wide technology policy and the Department of Veterans Affairs. Prior to joining FCW, Mazmanian was technology correspondent for National Journal and served in a variety of editorial roles at B2B news service SmartBrief. Mazmanian has contributed reviews and articles to the Washington Post, the Washington City Paper, Newsday, New York Press, Architect Magazine and other publications.

Click here for previous articles by Mazmanian. Connect with him on Twitter at @thisismaz.

# data[].attributes.attachment_link.description
FOSE The proliferation of government data sets is providing developers with ample fodder for writing useful and potentially profitable applications around census, weather, health, energy, business, agricultural and other information. But as the government makes more and more data discoverable and machine readable, there is the threat that disparate threads can be pieced together in a way that yields information that is supposed to be private.

# data[].attributes.attachment_link.readable_text
Article has been originally published at Techeries.com by Shantanu Chaturvedi

The world is growing at an enormous speed every moment, and at the same speed is growing the overall data size across the globe, which technically constitutes the term known as 'Big Data'. Apart from big data, a yet another technological revolution that is taking the world by storm these days is 'Artificial Intelligence' or AI and 'Machine Learning'.

In simple terms, AI and machine learning is a set of technologies that empower connected machines and computers to learn, evolve and improve upon their own learning by reiterating and consistently updating the data bank through recursive experiments and human intervention.

As a matter of fact, the technological giants, corporations, and data scientists worldwide are already foreseeing big data to make a huge difference in the overall AI and machine learning landscape. In the annual Big Data Executive Survey conducted by NewVantage Partners this year, 88.5% of top executives surveyed were found saying that AI is going to be the most dominant factor that will have a disruptive impact on their companies. Let's have a look how is it going to realize in practical terms.

Big Data to Enhance Artificial Intelligence

Inherently, machine learning is defined as an advanced application of AI in interconnected machines and peripherals by granting them access to databases and making them learn new things from it on their own in a programmed manner. As the size of big data is continuously growing and new grounds are being broken in analyzing its implications as well, it is becoming more meaningful and contextually relevant for the machines to have a better idea of their functions with the help of big data analysis.

For example, the automation infrastructure of a leather garments plant based in Bangladesh that exports its products to the entire European market will be able to judge market requirements for the coming winter season in much accurate and insightful manner if it is able to access and analyze big data reports about the market, financial and weather conditions of that area throughout the year.

Big Data to Help Expand AI & Machine Learning Workforce

One of the major concerns people are having about AI today is that it will minimize human requirement in all the job sectors as most of the work will be done by robots and AI-based computers in future, while the truth is far from it when observed with the role to be played by big data in the picture. The sentimental and emotional big data analysis will always require human intelligence as the machines lack emotional intelligence and decision-making abilities based on sentiments.

For example, a data scientist analyzing the big data pertaining to a pharmaceutical giant that caters to the needs of South-East Asian market may be able to sense the pharmaceutical prescriptions to be launched in those areas keeping the local inhibitions and reservations in mind, while computer-based big data analysis can never yield such contextually sensitive search results.

Hence, the increasing collaboration between AI, machine learning, and big datawill only make way for talented and capable human data scientists to consistently evolve and rise in the market, and by all means, they will be required in a much huge number as the applications of these technologies gain movement.

AI & Machine Intelligence Solution Providers to Benefit from Big Data

Right now, the overall size of global markets for machine learning and artificial intelligence based solutions is highly limited. With increasing proliferation of big data analysis into the artificial intelligence and machine learning procedures, devices and machines will get smarter and able to perform in a better manner. This will lead to consistent improvement, enhancement and advancement in AI solutions which will boost the market adoption of these solutions, giving rise to a high increase in their market demand.

In his analytical piece, James Canton, a well-known Big Data, and AI expert suggests that as in near future even the network nodes, chips, sensors and the software programs that will run IoT networks will be AI enabled via the cloud or at the chip or infrastructure level, it will be impossible for the massive global IoT network of tomorrow that will process, distribute, collect big data to enable us, humans, to run this network without a digital brain or AI based solution that is smart enough to do this.

For example, big data analysis of AI best learning in the schools of rural and smaller towns of Latin American regions ten years hence will help inconsistent modification and revision of AI-based academic solutions, so that more schools and educational institutions will be encouraged to adopt artificial intelligence into their training and teaching methods and the market sector will witness a considerable growth accordingly.

Big Data to help in Global Diversification of AI &Machine Learning Offerings

With the advent of newer technologies and increasing scales of production, the prices of AI and machine learning based solutions will fall considerably. This will result in a massive adoption of these devices in third world nations which have diverse cultural, religious, ethnic, linguistic and political affiliations and inclinations. Therefore, the artificial intelligence and machine learning solutions to be provided in these markets will have to be trained and skilled differently from each other.

Banking upon big data analysis of these factors from region to region, the artificial intelligence and machine learning solutions will be able to be relevantly molded and trained so that the sentiment of different demographics don't get hurt, and these solutions also prove to be useful to the customers. For example, a women-centric artificial intelligence solution catering to the needs of Iranian women will have to be designed, developed and trained in an entirely different manner than the same being targeted upon the female customers in the Sri Lankan market.

Big Data backed Market Analysis to Boost AI & Machine Learning Adoption

Right now, the market sector of artificial intelligence and machine learning based products is in its nascent stage. As the market will grow, players will come to realize what sorts of product offerings, features and functionality are preferred by customers to have or not have on their devices and solutions. Here again, big data will come as a formidable rescue to the enterprises developing and selling AI and machine learning based solutions.

A market-wide and deep big data analysis focusing upon the sales patterns, feedback, and suggestions gathered from a large section of customers will allow the companies to develop solutions and services that are found to be more suitable according to the requirements of their customers.

For instance, an AI-based solution predicting forecasts in the commodity exchange in India will be required to undergo enormous modifications and improvements to fulfil the requirements of a large section of sellers and buyers, owing to the huge number of commodities varying from each other in terms of demands in local markets, seasons of availability and the control prices

Article has been originally published at Techeries.com by Shantanu Chaturvedi

# data[].attributes.attachment_link.title
How Big Data Is Empowering AI and Machine Learning?

# data[].attributes.attachment_link.description
Article has been originally published at Techeries.com by Shantanu Chaturvedi The world is growing at an enormous speed every moment, and at the same speed is growing the overall data size across the globe, which technically constitutes the term known as 'Big Data' .

# data[].attributes.attachment_link.description
PyData began as a workshop held in March of 2012 at the Googleplex in Mountain View, CA. Travis Oliphant and Peter Wang, from Continuum Analytics, along with Julie Steele and Edd Dumbill, from O'Reilly, organized the event. Many prominent individuals in the community were on-hand to deliver tutorials and discuss the current state of the Python Scientific Stack.

# data[].attributes.body
Datavisualisatie van de populairste pizza chains in de Verenigde Staten.

# data[].attributes.body
Naar bron

# data[].attributes.headline
16 mooie voorbeelden van data dashboards

# data[].attributes.subject
16 mooie voorbeelden van data dashboards

# data[].attributes.attachment_link.readable_text
Inspiratie voor het ontwikkelen van data dashboards

Het gebruik van data is natuurlijk niets nieuws. Wel is er gelukkig steeds meer aandacht voor de interpretatie van data. Datavisualisatie is er op gericht om informatie zo optimaal mogelijk te communiceren zodat marketeers, analisten en andere gebruikers op basis van die informatie beslissingen kunnen maken.

Data dashboards zijn een mooie manier om op continue basis de belangrijkste informatie naar betrokkenen te communiceren. Dat is moeilijker dan het lijkt; vaak worden dashboards volgepropt met metrics en wordt er niet goed genoeg gekeken naar welke informatie écht nodig is.

Geïnspireerd door een blog op Econsultancy In deze blog heb ik enkele gave voorbeelden van data dashboards op een rijtje gezet.

De (vernieuwde) Google Analytics app

Sinds afgelopen week ziet de Google Analytics app voor Android er heel anders goed uit.

Morning

Morning is een app met o.a. informatie over het weer, nieuws en to-do's. De interface is ontworpen door Chris Masterson.

Meer flat design

Helaas ook fictief. Van de hand van Riki Tanone.

Fitbit

Fitbit is een app waarmee je, met behulp van een armbandje, je gezondheid en beweging kunt bijhouden. Naast de app heeft Fitbit een erg mooi webdashboard.

Nektar

De glimps die ik af en toe opvang van Nektar is erg indrukwekkend. Helaas is deze tool nog altijd in ontwikkeling.

Start

Start is een template voor het cms van Wordpress, maar bevat ook visualisatie van data. En die ziet er dan ook heel goed uit.

Simyo

De informatie over het gebruik van telecomproviders is de laatste 2 jaar gelukkig flink verbeterd. Eén van de mooiere voorbeelden is de website van Simyo.

C*Tools

FoxMetrics

Analytics-tool FoxMetrics biedt gesegmenteerde analytics dashboards. Onderstaand screenshot is van het e-commerce dashboard (door Behzad Ghaffarian).

Aquicore

Aquicore is een tool voor energiemanagement. Zo kan je er precies mee bijhouden hoeveel apparaten gebruiken. Screenshot door Tyler Allen.

Statisti.cs

Ook een mooi voorbeeld van een dashboard voor energiemonitoring. Het is een fictief voorbeeld en is gemaakt door Derek Bender.

Geckoboard

Er zijn meerdere tools om zelf data dashboards te maken. Ik vind Geckoboard zelf één van de mooiere oplossingen.

Analytiks

Analytiks is een analytics app voor iPhone.

Oblivion

De uitdaging van actionable data dashboards is dat ze enerzijds veel informatie moeten communiceren, maar anderzijds ook begrijpelijk blijven. De computer van de control centers die ik afgelopen weekend in de film Oblivion zag is erg druk, maar heeft ook een duidelijke focus op de belangrijkste elementen.

Toutapp

Tweetping

Of het actionable is, mwah, maar awesome is het wel. Dichterbij werkelijke realtime datavisualisatie kom je niet. Pingdom visualiseert Twittergebruik naar geolocatie. Het resultaat is zeer indrukwekkend.

Weet jij mooie dashboards of sites die hierin hadden moeten staan? Laat het weten in de comments.

# data[].attributes.attachment_link.title
16 mooie voorbeelden van data dashboards

# data[].attributes.attachment_link.description
Inspiratie voor het ontwikkelen van data dashboards Het gebruik van data is natuurlijk niets nieuws. Wel is er gelukkig steeds meer aandacht voor de interpretatie van data. Datavisualisatie is er op gericht om informatie zo optimaal mogelijk te communiceren zodat marketeers, analisten en andere gebruikers op basis van die informatie beslissingen kunnen maken.

# data[].attributes.attachment_link.readable_text
The most challenging obstacle facing companies is recognizing if they have a valid use case requiring the use of Big Data technologies. Having a large amount of data does not necessitate the deployment of a Big Data solution architecture. A company should only determine to use a Big Data solution based on the characteristics and its intended use. This allows actionable insights to be more readily derived.

Other obstacles include the lack of skilled resources to administer and construct the solution, which poses a risk of underestimating the effort to deploy the solution, as well as a sub-par architecture that would lead to performance issues. Two other problems are insufficient governance associated with affected business processes and organizational change management, and misunderstanding the limitations of Big Data technologies and its ever-expanding ecosystem.

Companies must also adjust to new technologies and toolsets that are continuously introduced. Some are very mature, while some are quite nascent. Making sure the correct toolsets are select and deployed can be very challenging. With this, businesses often try to tackle too much too soon - the size of the selected use case may be so large and complex that it never produces any actionable insights in a reasonable amount of time.

Overcoming Big Data obstacles
Make sure that the identified use case is appropriate for a Big Data solution. Highly structured and transaction-oriented data may be better served by a traditional DBMS. Big Data solutions are extremely well-suited for unstructured and semi-structured data. Structured data may also be stored and transformed as required and integrated with unstructured data to provide richer insights.

Companies should not skimp on training or hiring. Having skilled resources is critical for the timely delivery of Big Data solutions that are efficient and meet consumers' analytic requirements. This should include understanding the Big Data ecosystem and how tool sets are related to specific use cases as well as to each other. This will lead to the right tool for the right job.

Businesses should also focus on governance and the definition of processes for security, data storage, movement and the deployment of analytic artifacts. This will ensure that discovered insights remain relevant to the identified use cases and target audiences. This cannot be overemphasized: keep it simple, but challengin, meaning the use case should be manageable. Capabilities should be defined that can be delivered in 30-60-day windows (otherwise, interest will be lost). Select a use case that is fairly well-understood. This will provide continuous delivery of capabilities, allow easier validation of results and provide a learning opportunity.

Consider leveraging cloud-based solutions. Cloud vendors have solutions that can be deployed very quickly (minutes to hours). These solutions can provide dynamic scaling and/or scaling on-demand. When analytic requirements change, the solution can be scaled up as required allowing for more complex and deep insights (data visualization, machine learning).

Finally, if using a dedicated analytics tool, make sure that tool has native Big Data integration. This will facilitate the development of self-service analytics and insights and lead to an agiler decision-making process.

The future of Big Data
The future of Big Data is strong and fast from a growth perspective:

New types of data are constantly appearing: Big Data tools and technologies are evolving to keep up. For example, data associated with virtual assistants and AI (Artificial Intelligence) applications is growing at an astronomical rate. New ways of storing and retrieving this data are always being developed.

It is becoming easier to develop, deploy and interact with Big Data solutions. This indicates that the barriers to entry to the world of Big Data are being reduced.

# data[].attributes.attachment_link.title
Assessing the need for Big Data solution architecture - SD Times

# data[].attributes.attachment_link.description
The most challenging obstacle facing companies is recognizing if they have a valid use case requiring the use of Big Data technologies. Having a large amount of data does not necessitate the deployment of a Big Data solution architecture. A company should only determine to use a Big Data solution based on the characteristics and its intended use.

# data[].attributes.body
Dit zijn goede voorbeelden van datavisualisaties

# data[].attributes.attachment_link.description
r/dataisbeautiful: A place for visual representations of data: Graphs, charts, maps, etc. DataIsBeautiful is for visualizations that effectively convey information. **Aesthetics are an important part of information visualization, but pretty pictures are not the aim of this subreddit.**

# data[].attributes.attachment_link.title
Now this is proper mapporn. Flight routes of the world [1700x970] * r/MapPorn

# data[].attributes.attachment_link.description
8977 points and 442 comments so far on reddit

# data[].attributes.attachment_link.description
Facts are fun, but most are presented in boring and badly edited videos. The Infographics Show focuses on making animated motion infographic videos, made in a fun and entertaining way. Software that we use: Adobe Audition for voice recording, Adobe Illustrator for dealing with vectors, and Adobe After Effects for animation.

# data[].attributes.attachment_link.description
At TED@Cannes, Gary Wolf gives a 5-min intro to an intriguing new pastime: using mobile apps and always-on gadgets to track and analyze your body, mood, diet, spending -- just about everything in daily life you can measure -- in gloriously geeky detail.

# data[].attributes.body
Voorbeeld van een duidelijke infographic

# data[].attributes.body
Dit is een voorbeeld van een goede infographic.

# data[].attributes.body
Voorbeeld van een slechte infographic

# data[].attributes.attachment_link.readable_text
Big data, as its proponents have been saying for nearly a decade now, can bring big benefits: advertisements focused on what you actually want to buy, smart cars that can help you avoid collisions or call for an ambulance if you happen to get in one anyway, wearable or implantable devices that can monitor your health and notify your doctor if something is going wrong.

It can also lead to big privacy problems. By now it is glaringly obvious that when people generate thousands of data points every day - where they go, who they communicate with, what they read and write, what they buy, what they eat, what they watch, how much they exercise, how much they sleep and more - they are vulnerable to exposure in ways unimaginable a generation ago.

It is just as obvious that such detailed information, in the hands of marketers, financial institutions, employers and government, can affect everything from relationships to getting a job, and from qualifying for a loan to even getting on a plane. While there have been multiple expressions of concern from privacy advocates and government, there has been little action to improve privacy protections in the online, always connected world.

It was more than five years ago that the Obama administration published a blueprint for what it termed a Consumer Privacy Bill of Rights (CPBR), in February 2012. That document declared that, "the consumer privacy data framework in the U.S. is, in fact, strong ... (but it) lacks two elements: A clear statement of basic privacy principles that apply to the commercial world, and a sustained commitment of all stakeholders to address consumer data privacy issues as they arise from advances in technologies and business models."

Three years later, in February 2015, that blueprint became proposed legislation by the same name, but it was immediately attacked, both by industry groups, who says it would impose "burdensome" regulations, and by privacy advocates, who says it was riddled with loopholes. It never made it to a vote.

The CPBR declaration that the, "consumer privacy data framework in the U.S. is, in fact, strong ..." ironically came about a year before revelations by former NSA contractor Edward Snowden that the U.S. government was, in fact, spying on its citizens.

Beyond that, government hasn't been able to agree on other privacy initiatives. The so-called broadband privacy rules issued by the Federal Communications Commission (FCC) just before the 2016 election, which would have limited data collection by Internet service providers (ISPs), were repealed by Congress in March, before they took effect.

Susan Grant, director of consumer protection and privacy at the Consumer Federation of America (CFA), called it "a terrible setback," and says it would allow ISPs, "to spy on their customers and sell their data without consent." Others, however, have argued that putting limits on ISPs would still leave other online giants like Google free to collect and sell the data they collect, and consumers would see few, if any, benefits.

Given all that, it should be no surprise that experts say privacy risks are even more intense, and the challenges to protect privacy have become even more complicated. Organizations like the CFA, the Electronic Privacy Information Center (EPIC) and the Center for Democracy and Technology (CDT), along with individual advocates like Rebecca Herold, CEO of The Privacy Professor, have enumerated multiple ways that big data analytics, and resulting automated decision-making, can invade the personal privacy of individuals. They include:

1. Discrimination

EPIC declared more than three years ago, in comments to the U.S. Office of Science and Technology Policy that, "The use of predictive analytics by the public and private sector ... can now be used by the government and companies to make determinations about our ability to fly, to obtain a job, a clearance or a credit card. The use of our associations in predictive analytics to make decisions that have a negative impact on individuals directly inhibits freedom of association."

Since then, things have gotten worse, privacy advocates say. While discrimination is illegal, automated decision-making makes it more difficult to prove. "Big data algorithms have matured significantly over the past several years, along with the increasing flood of data from the nascent internet of things, and the ability to analyze these data using variants of artificial intelligence." says Edward McNicholas, global co-leader of the Privacy, Data Security, and Information Law Practice at Sidley Austin LLP. "But despite this technological growth, the legal protections have not advanced materially."

"I think the discussion around big data has moved beyond mere accusations of discrimination to larger concerns about automated decision-making," says Joseph Jerome, policy counsel at the CDT, who noted that it has been used, "to direct calls at call service centers, evaluate and fire teachers, and even predict recidivism."

Herold has been saying for years that big data analytics can make discrimination essentially "automated," and therefore more difficult to detect or prove. She says that is true, "in more ways than ever" today. "Big data analytics coupled with internet of things (IoT) data will be - and has already been - able to identify health problems and genetic details of individuals that those individuals didn't even know themselves," she says.

McNicholas believes, "the most significant risk is that it is used to conceal discrimination based on illicit criteria, and to justify the disparate impact of decisions on vulnerable populations."

2. An embarrassment of breaches

By now, after catastrophic data breaches at multiple retailers like Target and Home Depot, restaurant chains like P.F. Chang's, online marketplaces like eBay, the federal Office of Personnel Management that exposed the personal information of 22 million current and former federal employees, universities, and online services giants like Yahoo, public awareness about credit card fraud and identity theft is probably at an all-time high.

Unfortunately, the risks remain just as high, especially given the reality that billions of IoT devices in everything from household appliances to cars, remain rampantly insecure, as encryption and security guru Bruce Schneier, CTO at IBM Resilient, frequently observes in his personal blog.

[ Related: The 15 biggest security breaches of the 21st century]

3. Goodbye anonymity

It is increasingly difficult to do much of anything in modern life, "without having your identity associated with it," Herold says. She says even de-identified data does not necessarily remove privacy risks. "The standards used even just a year or two ago are no longer sufficient. Organizations that want to anonymize data to then use it for other purposes are going to find it increasingly difficult. "It will soon become almost impossible to effectively anonymize data in a way that the associated individuals cannot be re-identified," she says.

Besides being vulnerable to breaches, IoT device are a massive data collection engine of users' most personal information. "Individuals are paying for smart devices, and the manufacturers can change their privacy terms at a moment's notice," Jerome says. "It's one thing to tell a user to stop using a web service; it's another to tell them to unplug their smart TV or disconnect their connected car."

4. Government exemptions

According to EPIC, "Americans are in more government databases than ever," including that of the FBI, which collects personally identifiable information (PII) including name, any aliases, race, sex, date and place of birth, Social Security number, passport and driver's license numbers, address, telephone numbers, photographs, fingerprints, financial information like bank accounts, and employment and business information.

Yet, "incredibly, the agency has exempted itself from Privacy Act (of 1974) requirements that the FBI maintain only, 'accurate, relevant, timely and complete' personal records," along with other safeguards of that information required by the Privacy Act, EPIC says. The NSA also opened a storage facility in Bluffdale, Utah, in 2014 that is reportedly capable of storing 12 zettabytes of data - a single zettabyte is the amount of information it would take 750 billion DVDs to store.

While there have been assurances, including from former President Obama, that government is "not listening to your phone calls or reading your emails," that obviously ducks the question of whether government is storing them.

5. Your data gets brokered

Numerous companies collect and sell consumer data that are used to profile individuals, without much control or limits. There was the famous case of companies beginning to market products to a pregnant woman before she had told others in her family, thanks to automated decision-making. The same can be true of things like sexual orientation or an illness like cancer.

"Since 2014, data brokers have been having a field day in selling all the data they can scoop up from anywhere they can find it on the internet. And there are few - none explicit that I know of - legal protections for involved individuals," Herold says. "This practice is going to increase, unfettered, until privacy laws restricting such use are enacted. There is also little or no accountability or even guarantees that the information is accurate.

Where do we go from here?

Those are not the only risks, and there is no way to eliminate them. But there are ways to limit them. One, according to Jerome, is to use big data analytics for good - to expose problems. "In many respects, big data is helping us make better, fairer decisions," he says, noting that it can be, "a powerful tool to empower users and to fight discrimination. More data can be used to show where something is being done in a discriminatory way. Traditionally, one of the biggest problems in uncovering discrimination is a lack of data," he says.

There is general agreement among advocates that Congress needs to pass a version of the CPBR, which called for consumer rights to include:

Individual control over what personal data companies collect from them and how they use it.

Transparency, or easily understandable and accessible information about privacy and security practices.

The collection, use and disclosure of personal data to be done in ways that are consistent with the context in which consumers provide the data.

Security and responsible handling of personal data.

Access to their personal data in usable formats, with the power to correct errors.

Reasonable limits on the personal data that companies collect and retain.

McNicholas says that "transparency" should include an overhaul of "privacy policies," which are so dense and filled with legalese that almost nobody reads them. "Telling consumers to read privacy policies and exercise opt-out rights seems to be a solution better suited to last century," he says. "Consumer privacy must shift to consumer-centric, where consumers have real control over their information."

Jerome agrees. "I certainly don't think we can expect consumers to read privacy policies. That's madness. What we should expect are better and more controls. It's a good thing that users can review and delete their Echo recordings. It's great that Twitter allows users to toggle all sorts of personalization and see who has targeted them," he says. "But ultimately, if individuals aren't given more options over collection and sharing, we're going to have serious issues about our personal autonomy."

Given the contentious atmosphere in Congress, there is little chance of something resembling the CPBR being passed anytime soon. That doesn't mean consumers are defenseless, however. What can they do?

Jerome says even if users don't read an entire policy, they should, "still take a moment before clicking 'OK' to consider why and with whom they're sharing their information. A recent study suggested that individuals would give up sensitive information about themselves in exchange for homemade cookies."

Herold offers several other individual measures to lower your privacy risks:

Quit sharing so much on social media. "If you only have a few people you want to see photos or videos, then send directly to them instead of posting where many can access them," she says.

Don't provide information to businesses or other organizations that are not necessary for the purposes for which you're doing business with them. Unless they really need your address and phone number, don't give it to them.

Use an anonymous browser, like Hotspot Shield or Tor (The Onion Router) when visiting sites that might yield information that could cause people to draw inaccurate conclusions about you.

Ask others not to share information online about you without your knowledge. "It may feel awkward, but you need to do it," she says, adding that the hard truth is that consumers need to protect themselves because nobody else will be doing it for them.

Regarding legislation, she says she has not heard about any other drafts of the CPBR in the works, "and I quite frankly do not expect to see anything in the next four years that will improve consumer privacy; Indeed, I expect to see government protections deteriorate. "I hope I am wrong," she says.

# data[].attributes.attachment_link.title
The 5 worst big data privacy risks (and how to guard against them)

# data[].attributes.attachment_link.description
Big data, as its proponents have been saying for nearly a decade now, can bring big benefits: advertisements focused on what you actually want to buy, smart cars that can help you avoid collisions or call for an ambulance if you happen to get in one anyway, wearable or implantable devices that can monitor your health and notify your doctor if something is going wrong.

# data[].attributes.headline
Verschil infographic en een datavisualisatie

# data[].attributes.subject
Verschil infographic en een datavisualisatie

# data[].attributes.attachment_link.readable_text
Bij een infographic draait het met name om wat de maker, of de afzender, wil vertellen, terwijl een datavisualisatie de lezer zelf het verhaal laat ontdekken. Een datavisualisatie vergt daarom wat meer van de lezer dan een infographic, waar het verhaal al is voorgekookt.

Hoewel datavisualisaties meestal voor de computer worden ontwikkeld en dynamisch en interactief zijn, kunnen datavisualisaties ook statisch ontwikkeld worden. Infographics zijn juist vaak statisch, maar kunnen ook interactief gemaakt worden, zie bijvoorbeeld deze over CO2 en het broeikaseffect van het PBL. Interactiviteit is daarom niet een eigenschap die aan een van beide disciplines is voorbehouden.

Wanneer laat je een infographic ontwikkelen?

Een datavisualisatie zul je gebruiken als je mensen wilt helpen bij het begrijpen en analyseren van een grote (complexe) dataset. Je hebt geen vooropgezet plan wat zij hier uit zouden moeten halen. Heb je wel een bepaalde boodschap of idee over het soort informatie dat mensen tot zich zouden moeten nemen, dan zul je eerder een infographic gaan maken.

We hopen dat hiermee wat meer duidelijkheid is geschept in het verschil tussen infographics en datavisualisaties. Kun jij je vinden in deze definitie, of hanteer jij een andere? Datavisualisatie is een relatief nieuwe discipline en verschillende mensen hanteren verschillende definities. We zijn benieuwd naar de jouwe!

Infographic of datavisualisatie laten ontwikkelen? Neem contact op!

# data[].attributes.attachment_link.title
Het verschil tussen infographics en datavisualisatie

# data[].attributes.attachment_link.description
Er is al veel geschreven over infographics en datavisualisatie, maar in de praktijk merk ik dat mensen nog steeds worstelen met wat nou precies het verschil is tussen die twee. Daarom zullen we het nog even kort proberen uit te leggen.

# data[].attributes.body
http://searchbusinessanalytics.techtarget.com/definition/big-data-analytics

# data[].attributes.headline
Wat is het verschil tussen een infographic en een datavisualisatie?

# data[].attributes.subject
Wat is het verschil tussen een infographic en een datavisualisatie?

# data[].attributes.body
Infographic gaat grotendeels om het laten zien van data, en dit op een manier waardoor het voor anderen makkelijk en duidelijk te zien valt hoe wat en wanneer. Een goede infogprahic verteld een verhaal die snel voor iedereen duidelijk is. Een datavisualisatie aan de andere kant gaat meer om het vergelijken van de datasets, hierbij kan je denken aan trends of dingen waarover mensen speculeren. Een voorbeeld te noemen: Door chocola leef je langer. Een datavisualisatie laat dan een grafiek met chocola inname zien, en eentje met de leeftijd die de persoon behaalt heeft. En dan is de vraag of hierin een trend valt te zien.

# data[].attributes.body
Top 25 Big Data bedrijven

# data[].attributes.attachment_link.readable_text
Big Data companies come in many different shapes and flavors. In fact you might say, a list of Big Data companies necessarily contains vendors with highly contrasting strategies - clearly, the data analytics market is in rapid flux.

Standards? Kind of. Not exactly. Depends who you ask.

It has been just seven years since Yahoo introduced Hadoop but the concept behind it, Big Data, has exploded in popularity as more and more firms launch pilot programs to gain insight from the massive amounts of data at their disposal.

Big Data has matured differently than most technologies, however. First, no one leader has emerged after nearly a decade. The analytics industry is still in growth mode, and leaders emerge when an industry consolidates.

Secondly, the big names got in the market early in a big way. That's also unprecedented, because established vendors have traditionally been notoriously slow to embrace a new technology. But already, IBM, Microsoft, SAP, HP, and Oracle are in the game.

So, which tools and platforms should you choose? Here are 25 of the top companies to consider in the Big Data world.
Please note: this list is NOT a ranking - the strategies are too different. So company number 7, for instance, is not a "better" Big Data vendor than company number 20. The many Big Data companies on this list offer approaches that focus on many different IT sectors.
Big Data Companies: The Leaders

Originally spun out of Stanford University as a research project, Tableau started out by offering visualization techniques for exploring and analyzing relational databases and data cubes and has expanded to include Big Data research. It offers visualization of data from any source, from Hadoop to Excel files, unlike some visualization products that only work with certain sources, and works on everything from a PC to an iPhone.
See user reviews of Tableau.
New Relic uses a SaaS model for monitoring Web and mobile applications in real-time that run in the cloud, on-premises, or in a hybrid mix. It uses more than 50 plug-ins from technology partners to connect to its monitoring dashboard. The plug-ins include PaaS/cloud services, caching, database, Web servers and queuing. Its Insights software for analysis works across the entire New Relic product line, and the company offers a product called Insights Data Explorer that is designed to make it easier for everyone on a software team to explore Insights events.
See user reviews of New Relic APM.
Alation crawls an enterprise to catalog every bit of information it finds and then centralizes the organization's knowledge of data, automatically capturing information on what the data describes, where the data comes from, who's using it and how it's used. In other words, it turns all your data into metadata, and allows for fast searches using English words and not computer strings. The company's products provide collaborative analytics for faster insight, a unified means of search, provides a more optimized data structure of the company's data, and assists in better data governance.
See user reviews of Alation.
Teradata has built a portfolio of Big Data apps into what it calls its Unified Data Architecture, which includes Teradata QueryGrid, Teradata Listener, Teradata Unity and Teradata Viewpoint. QueryGrid provides a seamless data fabric across new and existing analytic engines, including Hadoop. Listener is the primary ingestion framework for organizations with multiple data streams, Unity is a portfolio of four integrated products for managing data flow throughout the process, and Viewpoint is a custom Web-based dashboard of tools to manage the Teradata environment.
See user reviews of Teradata.
VMware has incorporated Big Data into its flagship virtualization product, called VMware vSphere Big Data Extensions. BDE is a virtual appliance that enables administrators to deploy and manage the Hadoop clusters under vSphere. It supports a number of Hadoop distributions, including Apache, Cloudera, Hortonworks, MapR and Pivotal.
See user reviews of VMware.
Splunk Enterprise started out as a log analysis tool but has since expanded its focus and now focuses on machine data analytics to make the information useable by anyone. It can monitor online end-to-end transactions, study customer behavior and usage of services in real time, monitor for security threats, and identify spot trends and sentiment analysis on social platforms.
See user reviews of Splunk.
Besides its mainframe and Power systems, IBM offers cloud services for massive compute scale through its Softlayer subsidiary. On the software side, its DB2, Informix and InfoSphere database software all support Big Data analytics and Cognos and SPSS analytics software specialize in BI and data insight. IBM also offers InfoSphere, the basic platform for building data integration and data warehousing used in a BD scenario.
See user reviews of IBM Cognos.
Formerly known as WebAction, Striim is a real-time, data streaming analytics software platform that reads in data from multiple sources such as databases, log files, applications and IoT sensors and allows customers to react instantly. Enterprises can filter, transform, aggregate and enrich data as it is coming in, organizing it in-memory before it ever lands on disk.

SAP's main Big Data tool is its HANA in-memory relational database, which the company says can run analytics on 80 terabytes of data and integrates with Hadoop. Although HANA is a row-and-column database, it can perform advanced analytics, like predictive analytics, spatial data processing, text analytics, text search, streaming analytics, and graph data processing and has ETL (Extract, Transform, and Load) capabilities.

While some companies specialize in one or few sources of data, SAP deals with data from a wide range of sources, including data from sensors, machine logs and other equipment; human generated data - social, point of sale (POS), ERP, emails documents and other things that make up enterprise data.
See user reviews of SAP Data Governance.
A creation of Greenplum employees, Alpine Data Labs puts an easy-to-use advanced analytics interface on Apache Hadoop to provide a collaborative, visual environment for building analytics workflow and predictive models that anyone can use, rather than requiring a high-priced data scientist to program the analytics.

Oracle has its Big Data Appliance that combines an Intel server with a number of Oracle software products. They include Oracle NoSQL Database, Apache Hadoop, Oracle Data Integrator with Application Adapter for Hadoop, Oracle Loader for Hadoop, Oracle R Enterprise tool, which uses the R programming language and software environment for statistical computing and publication-quality graphics, Oracle Linux and Oracle Java Hotspot Virtual Machine.
See user reviews of Oracle Big Data Appliance.
Calling itself the leader in self-service data analytics, Alteryx's software is meant for the business user and not the data scientist. It allows them to blend data from multiple and potentially disparate sources, analyze it and share it so that actions can be taken. Queries can be made from anything from a history of sales transactions to social media activity.
See user reviews of Alteryx.
Splice Machine bills itself as the provider of the only Hadoop relationship database management system (RDBMS). It can act as a general-purpose database that can replace Oracle, MySQL or SQL Server databases for various workloads on Hadoop. The latest version, 2.0, added Spark, which does all analytics in memory instead of on disk. Version 2.0 also added the ability to route work to one of two processing engines either OLTP or OLAP.
See user reviews of Splice Machine.
Pentaho is a suite of open source-based tools for business analytics that has expanded to cover Big Data. The suite offers data integration, OLAP services, reporting, a dashboard, data mining and ETL capabilities.

Pentaho for Big Data is a data integration tool based specifically designed for executing ETL jobs in and out of Big Data environments such as Apache Hadoop or Hadoop distributions on Amazon, Cloudera, EMC Greenplum, MapR, and Hortonworks. It also supports NoSQL data sources such as MongoDB and HBase. The company was acquired by Hitachi Data Systems in 2015 but continues to operate as a separate subsidiary.
See user reviews of Pentaho.
SiSense sells its Prism to the largest enterprises and some SMBs alike because of its small ElastiCube product, a high-performance analytical database tuned specifically for real-time analytics. ElastiCubes are super-fast data stores that are specifically designed for extensive querying. They are positioned as a cheaper alternative to HP's Vertica systems.
See user reviews of SiSense.
Thoughtworks incorporates Agile software development principals into building Big Data applications through its Agile Analytics product. Agile Analytics helps companies build applications for data warehousing and business intelligence using the fast paced Agile process for quick and continuous delivery of newer applications to extract insight from data.
See user reviews of Thoughtworks.
Tibco's Jaspersoft subsidiary has introduced an hourly offering on Amazon's Cloud where you can buy analytics starting at $0.48 per hour. The company is also big on embedded its analytics - having done so with 130,000 production applications worldwide, used by organizations such as Red Hat, CA, Verizon, Tata, Groupon, British Telecom, Virgin, and the U.S. Navy.
See user reviews of Tibco Jaspersoft.
Amazon has a number of enterprise Big Data platforms, including the Hadoop-based Elastic MapReduce, Kinesis Firehose for streaming massive amounts of data into AWS, Kinesis Analytics to analyze the data, DynamoDB big data database, NoSQL and HBase, and the Redshift massively parallel data warehouse. All of these services work within its greater Amazon Web Services offerings.

Most significant, AWS is attempting to woo legacy database customers to its newer offering. Experts disagree on how successful AWS will be in this effort, but it is clearly a highly aggressive competitive move.
See user reviews of Amazon Web Services.
Microsoft's Big Data strategy is fairly broad and has grown fast. It has a partnership with Hortonworks and offers the HDInsights tool based for analyzing structured and unstructured data on Hortonworks Data Platform. Microsoft also offers the iTrend platform for dynamic reporting of campaigns, brands and individual products. SQL Server 2016 comes with a connector to Hadoop for Big Data processing, and Microsoft recently acquired Revolution Analytics, which made the only Big Data analytics platform written in R, a programming language for building Big Data apps without requiring the skills of a data scientist.
See user reviews of Microsoft BI.
Google continues to expand on its Big Data analytics offerings, starting with BigQuery, a cloud-based analytics platform for quickly analyzing very large datasets. BigQuery is serverless, so there is no infrastructure to manage and you don't need a database administrator, it uses a pay-as-you-go model.

Google also offers Dataflow, a real time data processing service, Dataproc, a Hadoop/Spark-based service, Pub/Sub to connect your services to Google messaging, and Genomics, which is focused on genomic sciences.
See user reviews for Google Data Studio.
Mu Sigma offers an analytics services framework that looks at tables and tables and answers questions for the firm on issues like improved sales and marketing. It cleans up client data to show only relevant data, uses the data to understand it, generates insights from it and gives recommendations to the client. Mu Sigma tries to understand how the business actually works and then identifies where the problem actually is.

HP Enterprise has built up a considerable portfolio of Big Data products in a very short time. Its main product is the Vertica Analytics Platform, designed to manage large, fast-growing volumes of structured data and provide very fast query performance on Hadoop and SQL Analytics for petabyte scalability.

HPE IDOL software provides a single environment for structured, semi-structured and unstructured data. It supports hybrid analytics leveraging statistical techniques and Natural Language Processing (NLP).

HPE has a number of hardware products, including HPE Moonshot, the ultra-converged workload servers, the HPE Apollo 4000 purpose-built server for Big Data, analytics and object storage. HPE ConvergedSystem is designed for SAP HANA workloads and HPE 3PAR StoreServ 20000 stores analyzed data, addressing existing workload demands and future growth.
See user reviews of HPE Big Data Solutions.
BigPanda offers a data science algorithm-based platform specifically for IT and DevOps staff that is specifically geared toward addressing alert overload. One of the many sources of Big Data is logs, and they can quickly get out of hand with redundant or false alerts. The company noticed that developers were being overwhelmed with alerts from their logs and had no idea which were real and which were false flags. BigPanda filters down that overload to just the meaningful alerts, allowing IT to react quicker to real problems.
See user reviews of Big Panda.
A highly vertical but important service, Cogito Dialog uses behavioral analytics technology, including analysis of everything from customer emails to social media to analysis of the human voice, to help phone support personnel improve their communications while on the phone with customers and to help organizations better manage agent performance.
See user reviews of Cogito.
Datameer claims its end-to-end data analytics solution for Hadoop enables business users to discover insights in any data via wizard-based data integration, iterative point-and-click analytics, and drag-and-drop visualizations, regardless of the data type, size, or source.
See user reviews of Datameer.

# data[].attributes.attachment_link.description
Big Data companies come in many different shapes and flavors. In fact you might say, a list of Big Data companies necessarily contains vendors with highly contrasting strategies - clearly, the data analytics market is in rapid flux. Standards? Kind of. Not exactly. Depends who you ask.

# data[].attributes.body
link met tools

# data[].attributes.attachment_link.readable_text
Data isn't a thing that's easy for the average person to grasp. While some can look through a spreadsheet and instinctively find the information they need within a mass of figures, the rest of us need a little help, and that's where data visualisation can be a real help.

For the designer, the challenge is not only in rendering a set of data in an informative way, but also in presenting it so that it that stands out from the mass of competing data streams.

One of the best ways to get your message across is to use a visualisation to quickly draw attention to the key messages, and by presenting data visually, it's also possible to uncover surprising patterns and observations that wouldn't be apparent from looking at stats alone. And nowadays, there's plenty of free graphic design software to help you do just that.

As author, data journalist and information designer David McCandless said in his TED talk: "By visualising information, we turn it into a landscape that you can explore with your eyes, a sort of information map. And when you're lost in information, an information map is kind of useful."

There are many different ways of telling a story, but everything starts with an idea. So to help you get started we've rounded up some of the most awesome data visualisation tools available on the web.

Make charts, presentations and dashboards with this flexible software. You can perform your analysis using JavaScript, Puthon, R, Matlab, Jupyter or Excel, and there are several options for importing data. The visualisation library and online chart creation tool allow you to make great-looking graphics.

DataHero enables you to pull together data from cloud services and create charts and dashboards. No technical abilities are required, so this is a great tool for your whole team to use.

Although armed with only six chart types, open source library Chart.js is the perfect data visualisation tool for hobbies and small projects. Using HTML 5 canvas elements to render charts, Chart.js creates responsive, flat designs, and is quickly becoming one of the most poplar open-source charting libraries. Read more about how to use Chart.js here.

Packed with graphs, charts, maps and more, Tableau Public is a popular data visualisation tool that's also completely free. Users can easily drag and drop data into the system and watch it update in real-time, plus you can collaborate with other team members for quick project turnaround.

Open, customisable, and free to download and modify, RAWGraphs lets users create vector-based data visualisations. Data can be safely uploaded from apps to computers, plus it can be exported as an SVG or PNG and embedded in your webpage.

Dygraphs is a fast, flexible open source JavaScript charting library that allows users to explore and interpret dense data sets. It's highly customisable, works in all major browsers, and you can even pinch to zoom on mobile and tablet devices.

ZingChart is a JavaScript charting library and feature-rich API set that lets you build interactive Flash or HTML5 charts. It offers over 100 chart types to fit your data.

If you're looking for a data viz tool with mapping, InstantAtlas is worth checking out. This tool enables you to create highly-interactive dynamic and profile reports that combine statistics and map data to create engaging data visualisations.

Modest Maps is a lightweight, simple mapping tool for web designers that makes it easy to integrate and develop interactive maps within your site, using them as a data visualisation tool.

The API is easy to get to grips with, and offers a useful number of hooks for adding your own interaction code, making it a good choice for designers looking to fully customise their user's experience to match their website or web app. The basic library can also be extended with additional plugins, adding to its core functionality and offering some very useful data integration options.

Another mapping tool, Leaflet makes it easy to use OpenStreetMap data and integrate fully interactive data visualisation in an HTML5/CSS3 wrapper.

The core library itself is very small, but there are a wide range of plugins available that extend the functionality with specialist functionality, such as animated markers, masks and heatmaps. It's perfect for any project where you need to show data overlaid on a geographical projection.

Billed as a "computational knowledge engine", the Google rival WolframAlpha is really good at intelligently displaying charts in response to data queries, without the need for any configuration. If you're using publicly available data, this offers a simple widget builder to make it really simple to get visualisations on your site.

Visualize Free is a hosted tool that allows you to use publicly available datasets, or upload your own, and build interactive visualisations to illustrate the data. The visualisations go well beyond simple charts, and the service is completely free, plus while development work requires Flash, output can be done through HTML5.

Orientated towards making positive change to the world, Better World Flux has some lovely visualisations of some pretty depressing data. It would be very useful, for example, if you were writing an article about world poverty, child undernourishment or access to clean water. This tool doesn't allow you to upload your own data, but does offer a rich interactive output.

FusionCharts Suite XT brings you 90+ charts and gauges, 965 data-driven maps, and ready-made business dashboards and demos. FusionCharts comes with extensive JavaScript API that makes it easy to integrate it with any AJAX application or JavaScript framework. These charts, maps and dashboards are highly interactive, customisable and work across all devices and platforms.

Another jQuery plugin, jqPlot is a nice solution for line and point charts. It comes with a few nice additional features such as the ability to generate trend lines automatically, and interactive points that can be adjusted by the website visitor, updating the dataset accordingly.

D3.js is a JavaScript library that uses HTML, SVG, and CSS to render some amazing diagrams and charts from a variety of data sources. This library, more than most, is capable of some seriously advanced visualisations with complex data sets. It's open source, and uses web standards so is very accessible. It also includes some fantastic user interaction support.

A fantastic library written by Nicolas Belmonte, the JavaScript InfoVis Toolkit includes a modular structure, allowing you to only force visitors to download what's absolutely necessary to display your chosen data visualisations. This library has a number of unique styles and swish animation effects, and is free to use (although donations are encouraged).
Next page: More top tools for data visualisation...

# data[].attributes.attachment_link.title
The 35 best tools for data visualization

# data[].attributes.attachment_link.description
Take the hard work out of creating charts and infographics with these tools.

# data[].attributes.attachment_link.title
United States gun death data visualization by Periscopic

# data[].attributes.attachment_link.description
Updated for the 2020 U.S. election, the most recent data on US gun killings: 472,332 stolen years. A look at law enforcement reported gun deaths - by @periscopic http://guns.periscopic.com/

# data[].attributes.attachment_link.title
The world mapped by Street View coverage [8192x8192][OC] * r/MapPorn

# data[].attributes.attachment_link.description
11000 points and 505 comments so far on reddit

# data[].attributes.attachment_link.description
Artist, designer, programmer, and entrepreneur specializing in data and digital technologies.

# data[].attributes.attachment_link.title
I tried to find the date of origin for every international border [OC] [5446x6217] * r/MapPorn

# data[].attributes.attachment_link.description
22762 points and 622 comments so far on reddit

# data[].attributes.attachment_link.description
Distilling the world’s data, information & knowledge into beautiful infographics & visualizations

# data[].attributes.body
Wat de meeste infographics met elkaar verbindt is dat ze een verhaal vertellen. Een infographic bevat een kop, een midden en een staart. De lezer wordt als het ware door het verhaal heen geleid.
Een infograpic bestaat vaak uit een duidelijke titel, een aantal grafieken en iconen die verschillende aspecten van een verhaal laten zien, begeleid van korte teksten, en een conclusie of boodschap.

Een datavisualisatie, ook wel bekend als informatievisualisatie, vertelt daarentegen geen vooropgezet verhaal. Een datavisualisatie kun je eigenlijk zien als een dataset, die beeldend is gemaakt. Dus in plaats van een rij getallen, zie je lijnen, staafdiagrammen of cirkels met labels. Deze data kun je visueel analyseren en vaak, als de datavisualisatie interactief is, exploreren

# data[].attributes.attachment_link.readable_text
Bij een infographic draait het met name om wat de maker, of de afzender, wil vertellen, terwijl een datavisualisatie de lezer zelf het verhaal laat ontdekken. Een datavisualisatie vergt daarom wat meer van de lezer dan een infographic, waar het verhaal al is voorgekookt.

Hoewel datavisualisaties meestal voor de computer worden ontwikkeld en dynamisch en interactief zijn, kunnen datavisualisaties ook statisch ontwikkeld worden. Infographics zijn juist vaak statisch, maar kunnen ook interactief gemaakt worden, zie bijvoorbeeld deze over CO2 en het broeikaseffect van het PBL. Interactiviteit is daarom niet een eigenschap die aan een van beide disciplines is voorbehouden.

Wanneer laat je een infographic ontwikkelen?

Een datavisualisatie zul je gebruiken als je mensen wilt helpen bij het begrijpen en analyseren van een grote (complexe) dataset. Je hebt geen vooropgezet plan wat zij hier uit zouden moeten halen. Heb je wel een bepaalde boodschap of idee over het soort informatie dat mensen tot zich zouden moeten nemen, dan zul je eerder een infographic gaan maken.

We hopen dat hiermee wat meer duidelijkheid is geschept in het verschil tussen infographics en datavisualisaties. Kun jij je vinden in deze definitie, of hanteer jij een andere? Datavisualisatie is een relatief nieuwe discipline en verschillende mensen hanteren verschillende definities. We zijn benieuwd naar de jouwe!

Infographic of datavisualisatie laten ontwikkelen? Neem contact op!

# data[].attributes.attachment_link.title
Het verschil tussen infographics en datavisualisatie

# data[].attributes.attachment_link.description
Er is al veel geschreven over infographics en datavisualisatie, maar in de praktijk merk ik dat mensen nog steeds worstelen met wat nou precies het verschil is tussen die twee. Daarom zullen we het nog even kort proberen uit te leggen.

# data[].attributes.body
Quantified Self is een beweging om technologie te laten integreren op aspecten van het dagelijks leven, in termen van inputs. Het gaat om het in kaart brengen van informatie over jezelf. Met deze informatie kun je patronen herkennen. Die patronen kun je vervolgens gebruiken om gezonder, slimmer, fitter en gelukkiger te worden.

# data[].attributes.body
link van uitleg

# data[].attributes.attachment_link.readable_text
In technologiekringen gaat een aforisme rond als er over Big Data wordt gesproken: als de berg mest maar groot genoeg is, vind je uiteindelijk wel een keer een gouden munt. Met andere woorden, als je genoeg data doorploegt, vind je altijd wel iets interessants.

Big Data - om onverklaarbare redenen altijd met hoofdletters geschreven - wordt het digitale goud, roepen de technologieprofeten. De datarevolutie zal onze economie en samenleving fundamenteel ontwrichten. Bestaande industrieën kwijnen weg (denk aan media, winkels, eigenlijk alles wat tussen producent en consument in staat). Wetenschap zal, volgens techno-extremisten, geen nood meer hebben aan theorieën. Als de datasets maar groot genoeg zijn, vinden we vanzelf verbanden tussen allerlei fenomenen. Iedereen met basale computervaardigheden kan straks kennis uit data halen. Iedereen gelijk. Iedereen blij.

Maar wat is Big Data precies?

Laat ik beginnen met een definitie. Die is er niet. Tenminste, er is geen definitie waar consensus over bestaat. De data kunnen van alles zijn als het maar digitaal is: video, audio, tekst, cijfers. Sommigen scharen alle data die niet meer op één pc passen onder Big Data. Anderen vinden dat data pas Big worden als ze het niveau van meerdere servers ontstijgen. Grote bedrijven hebben enorme serverfarms om hun online diensten te hosten of om hun dagelijkse operaties te kunnen uitvoeren. De wereld komt vol te hangen met sensoren - toegangspoorten, camera's, telefoons, meetapparatuur - die continu registreren en data opslaan, die wij vervolgens weer kunnen gebruiken. De digitale wereld en de fysieke wereld worden steeds meer één lichaam en data is het bloed.

Wat wel overeenkomt in deze definities is dat Big Data een omvang van een verzameling data beschrijft. Dit jaar zou 600 miljard gigabyte aan data door de leidingen van het internet gieren: lolcatz, netflix, het geklets op Twitter en Facebook. Wekelijks stroomt, volgens Cisco, een miljoen jaar aan videomateriaal over het net. Deeltjesversneller Large Hadron Collider produceert bijvoorbeeld één petabyte aan data per seconde (ongeveer 10.000 gigabyte) om dat ene ongrijpbare Higgs-boson, het god-deeltje, te vinden.

Maar steeds vaker hoor je dat Big Data geen omvang beschrijft, maar een ontwikkeling. Het bevat namelijk twee componenten. Allereerst de computertechnologie: de steeds geavanceerder hard- en software die het mogelijk maakt meer data te verzamelen, te bewerken en te bewaren. Het tweede component is de statistiek die het mogelijk maakt om in een verzameling losse data betekenis te vinden.

Is Big Data nieuw?

Nee. Grote bedrijven en wetenschapsinstellingen hebben altijd al veel data verzameld. De Engelse hoogleraar wiskunde en statistiek David Hand gaf recentelijk een lezing bij de Koninklijke Nederlandse Academie voor Wetenschappen in Amsterdam. Hij wees het publiek er fijntjes op dat een supermarktketen als Walmart al in de vroege jaren negentig 7 miljard transacties per jaar verwerkte en analyseerde. Mobil Oil gebruikte meer dan 100 terabytes aan data (een terabyte is bijna 1000 gigabyte) voor zijn operaties. Dus hoezo zou Big Data nieuw zijn?

Volgens hem en andere onderzoekers wordt de populariteit van Big Data vooral gedreven door slimme marketingjongens. Big Data apelleert aan het gevoel in een nieuwe wereld te zijn beland vol mogelijkheden. Een nieuwe wereld vol economische kansen. We laten zoveel datasporen achter, dat die marketingjongens kunnen zien wanneer ze ons het beste kunnen verleiden.

Is Big Data een hype?

Ja. Maar geen inhoudsloze. De twee componenten van Big Data (hard- en softwarecapaciteiten en statistiek) leiden tot een explosie van nieuwe toepassingen op het gebied van zogenoemde predictive analysis - dus: toekomstvoorspellingen.

Het meest aansprekende voorbeeld hiervan is warenhuis Target, een soort Amerikaanse V&D. Die kan op basis van haar enorme database met historisch en actueel koopgedrag inschatten of een klant zwanger is of niet. Het verhaal gaat Lees hier How Companies Learn Your Secrets dat Target een aanbieding voor kinderspullen stuurde naar een tienermoeder in spe en dat haar ouders er op die manier achterkwam dat ze zwanger was. Pijnlijk.

Big Data wordt vooral gebruikt om correlaties te vinden tussen fenomenen, personen en gebeurtenissen. Op basis van die correlaties worden vervolgens beslissingen genomen. De reclassering in Baltimore maakt bijvoorbeeld gebruik van risicoprofielen die voor iedere vrijgelaten gevangene de kans aangeven of die dader of slachtoffer wordt van een levensmisdrijf. Op basis daarvan wordt iemand meer of minder in de gaten gehouden.

Als je er oog voor hebt, zie je die toepassingen van Big Data overal. Verzekeraars gebruiken bergen informatie om risico's van individuele klanten in te schatten. Beurshandelaren voeden hun modellen met tientallen, soms duizenden datastromen om handelsrisico's en buitenkansjes vroeg te zien aankomen. Geheime diensten, denk aan de NSA, speuren in vele hooibergen naar mensen die in potentie - ze hebben het immers nog niet gedaan - een aanslag plegen.

De belofte van Big Data is dat die analyse steeds meer in het hier en nu plaatsvindt. Neem de nieuwe webservice Google Now. Die monitort zoveel mogelijk data rondom jouw persoon: wat je online doet, de inhoud van je e-mails, met wie je belt, waar je je bevindt en waar je naartoe gaat. Uit talloze andere online bronnen haalt Google informatie die voor jou relevant zou kunnen zijn. Het algoritme "weet" dat je vandaag op reis gaat en "weet" dat het vliegtuig vertraagd is. Zonder dat je hoeft te zoeken, krijg je een bericht hierover zodat je nog even thuis kunt relaxen. Een persoonlijke assistent die je door het dagelijks leven loodst. Dat is nog eens een belofte.

Dat klinkt indrukwekkend, maar ook een beetje eng. Tijd om ons zorgen te maken?

Ja en nee.

Ja, Big Data en dan vooral de toepassingen op het gebied van predictive analysis hebben de potentie om diep in ons leven te dringen. Daarnaast kan het ook tot discriminatie leiden. Steeds vaker zal je behandeld worden op basis van je risicoprofiel. Mag je ergens wel of niet door? Krijg je wel of geen lening? Als het profiel niet klopt, is het genomen besluit ook onjuist. En probeer je maar eens te verweren tegen een beslissing van een algoritme, als je überhaupt al weet dat er een profiel van je bestaat.

En nee. Er is nog steeds te veel hijgerigheid rondom Big Data. De moderne Big Data-sets gaan vooral over het nu en de nabije toekomst: transacties van de afgelopen jaren, tweets die sinds 2006 zijn gestuurd, Facebookposts die sinds 2005 zijn geplaatst, tolbetalingen op de autoweg sinds 2003. Ze zeggen weinig over wat ooit was. Als de data ouder zijn, komen er allemaal problemen bij die het moeilijk maken ze te analyseren. Definities veranderen over tijd - wie als arm wordt gezien bijvoorbeeld - waardoor je historische data niet met huidige data kunt vergelijken.

Meer Big Data is ook niet altijd beter. Twitter heeft bijvoorbeeld miljarden tweets beschikbaar. Daar kan je inhoudelijke analyses op loslaten, maar ook netwerken van mensen mee in kaart brengen. Maar hoe representatief is Twitter? Hoe veralgemeniseerbaar zijn die data? Vaak niet. Omvang, goede soft- en hardware en de beste statistiek nemen dit soort beperkingen niet weg.

Daarnaast treedt een effect op wat onderzoekers Danah Boyd en Kate Crawford apophenia noemen: maak de schaal groot genoeg en je vindt altijd wel iets interessants. Er is bijvoorbeeld een correlatie aangetoond tussen de resultaten van een American Football-team en de prestaties van de Amerikaanse economie. Maar niemand zal het in zijn hoofd halen een oorzakelijk verband te leggen.

Dat onze informatiemaatschappij steeds meer data driven wordt, daar twijfel ik niet aan. Dat die informatie-explosie ons dagelijks leven meer en meer zal beïnvloeden, neem ik zo aan. Maar de uitdaging voor Big Data-profeten wordt wel om aan te tonen dat er tussen die enorme hopen mest echt een gouden muntje ligt, en niet een paar koperen stuivers.

# data[].attributes.attachment_link.description
In technologiekringen gaat een aforisme rond als er over Big Data wordt gesproken: als de berg mest maar groot genoeg is, vind je uiteindelijk wel een keer een gouden munt. Met andere woorden, als je genoeg data doorploegt, vind je altijd wel iets interessants.

# data[].attributes.body
Tegenwoordig is life logging een movement waarbij de gemiddelde mens zijn leven bijhoud door middel van apps die data van het algemene leven opnemen.

Life logging wordt gedaan met wearables. Dit zijn makkelijk draagbare gadgets die apps bevatten. Met deze apps kun je bijhouden wat je doet op een dag.

# data[].attributes.body
Men spreekt van big data wanneer men werkt met een of meer datasets die te groot zijn om met reguliere databasemanagementsystemen onderhouden te worden. Big data spelen een steeds grotere rol. De hoeveelheid data die opgeslagen wordt, groeit exponentieel. Dit komt doordat consumenten zelf steeds meer data opslaan in de vorm van bestanden, foto's en films (bijvoorbeeld op Facebook of YouTube, waar Facebook de door de gebruikers gewiste data toch bewaart) en organisaties, overheden en bedrijven steeds meer data over burgers produceren en opslaan, maar ook doordat steeds meer apparaten zelf data verzamelen, opslaan en uitwisselen (het zogenaamde internet der dingen).

De definitie van big data is niet altijd duidelijk en de term big data wordt vaak ten onrechte gebruikt. Volgens Gartner gaat het om drie factoren:

de hoeveelheid data;

de snelheid waarmee de data binnenkomen en/of opgevraagd worden;

de diversiteit van de data. Hiermee wordt met name bedoeld dat de data ongestructureerd is en niet in een traditionele database opgeslagen kan worden.

Als aan minimaal twee van bovenstaande factoren is voldaan, spreekt men in het algemeen over big data.

Drie andere factoren zijn:

·        de variatie in de data. M.a.w. verschillende bronnen kunnen elkaar tegenspreken en het geheel extra compliceren;

·        de kwaliteit van de data: de ene bron is minder betrouwbaar dan de andere;

·        de complexiteit van de data: de mate waarin ongestructureerde data van verschillende bronnen met elkaar te combineren zijn.

# data[].attributes.body
Je kan quantified self apps downloaden om alle onderdelen van je leven te meten – van eten, slapen, bewegen, sporten, stress en doelen halen tot nog veel meer.

# data[].attributes.attachment_link.readable_text
Pre 1900s 1910 1920 1930 1940 1950 1960 1970 1980 1990 2000 Jump Blues Electric Blues Boogie woogie Blues Emo No Wave Gothic Rock New German Wave Post Punk Noise Rock Dark Wave Industrial Rock 2 Tone New Wave Alternative Country Gothabilly Country Pop Rockabilly Western Swing Country

# data[].attributes.attachment_link.description
Watch the development of rock music unfold and listen to it's musical influence over the past century.