# data[].attributes.body
Name1: Jim Buchan
Student ID 12345678

Name2: Albert Einstein
Student ID 1234567

# data[].attributes.attachment_link.description
In this InfoQ eMag, we curated some of the best serverless content into a single asset to give you a relevant, pragmatic look at this emerging space.

# data[].attributes.attachment_link.title
Cats, Qubits, and Teleportation: The Spooky World of Quantum Computation (Part One)

# data[].attributes.attachment_link.description
By the time most of us reach adulthood, we know a few basic truths: cats cannot be simultaneously alive and dead; objects at opposite ends of the universe can't affect each other; and computers operate on 0s and 1s, and that's the most fundamental unit of information.

# data[].attributes.attachment_link.description
John Chapin describes the nuts and bolts of building large-scale serverless applications using Java. He walks through the efficacy of a Java approach to a variety of serverless use cases, and through live examples demonstrates pragmatic approaches to developing scalable, evolvable, multi-Lambda applications.

# data[].attributes.attachment_link.readable_text
In a recent Hyperledger blog post, Jesse Chenard, CEO of MonetaGo, discusses how blockchain is poised to reinvent traditional Business Process Management platforms (BPM). A challenge with existing BPM platforms is that data is usually stored in organizational silos and challenges exist in counterparty transaction exchanges. A blockchain solution can provide auditing across boundaries, without leaking sensitive information to additional parties.

According to a Reuters article, it claims that the BPM market will be worth approximately $18 billion USD by 2022. Traditional BPM software provides workflow management solutions and have been around for decades. However, one of the constraints of this traditional approach is the data that is captured within these solutions becomes trapped within an organization's boundary, Chenard describes how this can be avoided:

Traditional BPM services tend to handle internal workflows within a single organization only. They do not manage the workflow process and information across organizations. If they do, they fall into the same trap that blockchain is helping to move away from, which is having a centralized repository of information controlled by a third party (in this case, the business process or workflow provider). If a third party holds data and is responsible for passing it from one organization to another, there is a risk of resiliency problems that come with that central or single point of failure.

However, introducing blockchain for the creation of peer-to-peer BPM systems has some advantages. Chenard explains:

Peer-to-peer BPM systems that eliminate the central repository of information and allows multiple corporations to exchange information directly with counterparties while guaranteeing the integrity of the process. The system allows for organizations to verify and enforce that specific steps are being taken and performed correctly by any party on the network.

An industry that can benefit from this type of blockchain architecture is financial services. The Reserve Bank of India recently licensed three entities to provide a lending solution, built on Hyperledger Fabric, that targets fraud reduction. RXIL, A.TReDS and M1xchange teamed up to build a receivables financing solution. Receivables financing is one of the fastest growing and efficient lending tools for small business to address their working capital needs. Within India, the financing market provides $219 billion USD to micro, small and medium businesses. Yet, there is more than $188 billion USD in unmet needs. Together, this group built a competitive marketplace to address this demand.

A challenge that this group sought to address was preventing organizations from seeking redundant funding by submitting applications for credit to multiple providers. Chenard shares how they were able to prevent duplicate funding:

By implementing a common blockchain platform, these exchanges were able to eliminate instances of double financing without sharing specific elements of any invoice or client. Ultimately, this implementation leads to the exchanges being able to offer better rates across the board to all of their customers and provide access to capital for more businesses otherwise deemed too risky.

Another aspect that needed to be addressed was the three organizations involved in building this platform are all competitors. As a result, maintaining privacy was really important. Chenard explains:

Clients are particularly sensitive about their sourcing inputs, and it is imperative that the exchanges not provide any of their client information to a shared registry controlled by any one entity. By creating a blockchain network built on Hyperledger Fabric, together with some intelligent cryptography, that concern is eliminated as the technology enables the exchanges to work together on a shared network to achieve shared goals without compromising privacy.

One of the benefits of using BPM, in a blockchain architecture, is participants and counterparties are still able to maintain control of their data without leaking customer data. This allows organizations to enforce network rules while still maintaining control.

While the project in India demonstrates that blockchain can be used in BPM architectures, there are still some challenges from a tooling and feature perspective. Chenard adds:

While saying blockchain technology as a communication layer for Business Process Management is a promising use case, it is by no means an out of the box solution - a lot depends on the actual workflow, the use case, and the local laws and regulations.

The BPM industry is very mature and approchable for developers and users, whereas blockchain continues to be an emerging technology. But, as demonstrated in the use case in India, blockchain can play an important role in business process management that spans multiple counterparties.

# data[].attributes.attachment_link.title
How Blockchain is Reinventing Business Process Management

# data[].attributes.attachment_link.description
In a recent Hyperledger post, Jesse Chenard discusses how blockchain is poised to reinvent traditional Business Process Management platforms. A challenge with existing BPM platforms is that data is usually stored in organizational silos and challenges exist in exchanging counterparty transactions. A blockchain solution can provide auditing across boundaries, without leaking sensitive information.

# data[].attributes.attachment_link.readable_text
Remember the movie "The Imitation Game"? The tragic story of a brilliant man who decrypted secret German Enigma messages, indirectly shortened World War II, saved millions of lives, and was later charged for homosexuality, forced to undergo chemical treatment, and ended his life shortly after?

The real Alan Turing accomplished many more brilliant miracles than this. He also published papers on theories of artificial intelligence (AI). In fact, the title "The Imitation Game" had little to do with the movie. It was a game he mentioned in one of his papers where humans will one day engineer a machine to imitate humans so well that a human on the other side of the room will be fooled he was communicating with another human. Turing was a pioneer in the field of computer science. Only after his death would he be known as the father of AI.

In the movie, he creates a machine that attempts to exhaust all possibilities of combinations of letters and words, trying to match the secret enigma code into meaningful words. The machine is on all day and all night non-stop. Imagine how long it'll take a human to look at all the possible patterns and try to decipher encrypted secret war messages. The machine couldn't have deciphered the code without Turing giving it instructions of what to do and how to do it and Turing wouldn't have been able to crack the code on his own either. It was the beautiful marriage of both a human brain and a machine's speed and accuracy.

Relationship Between AI, Machine Learning, and Deep Learning

Artificial intelligence (AI) is the field that seeks to study and design machines that are capable of human intelligence and decision making. Machine Learning (ML) is a subfield of AI that uses statistical mathematical approaches in an attempt to achieve AI.

Stanford University defines ML as "the science of getting computers to act without being explicitly programmed." Deep Learning (DL) is a subfield of ML that implements machine learning similar to the biology of our brains, taking in data of many dimensions and processing them through several networks and neural network layers.

That's tough to digest! Don't worry, I'll explain. To understand ML, let's understand it from a bigger scope of things, starting with AI.

Why Do We Have This Field and Why Does It Matter?

In the 1950's, modern day calculators and ATM bank machines would be considered AI since the act of calculating, withdrawing, and depositing money can be considered human intelligence. Fast forward to today, these tools are just "AI effects" because the definition of AI has evolved from just human intelligence to also demonstrating decision making

Imagine you wanted to go to the nearest coffee shop that's open right now at 5 a.m. Instead of looking at a collection of menus for the hours of operation and seeing which shop is closest, now we can ask voice assistants Alexa, Google Assistant, Siri, or Cortana,"What's the nearest coffee shop?"

Now imagine you watched a comedy and couldn't get enough of laughing. Instead of manually searching through movie titles, trying to figure out if the type of movie is a comedy, and then collecting a list of potential comedies to watch next, we can now rely on Netflix or YouTube to display a list of recommendations immediately after the movie finishes playing. Isn't it amazingly convenient that the next YouTube video that automatically plays is always a similar topic to the video you previously watched, given there are so many videos of different topics out there?

These are all examples of modern-day AI in progress. While an ATM machine can complete one task of giving me $100 when I press the withdraw button, self-driving cars can complete multiple spontaneous tasks that it decides is right: drive straight at a green light, slow down, turn, stop at the destination. To generalize, AI's carry out a list of tasks. The more tasks it can carry out, the more intelligent it seems.

"AI is the new electricity" - Andrew Ng

With this ambitious goal to empower humans, AI is broken into smaller more specialized subfields. Machine Learning is the most essential subfield. An approach to achieving AI, we teach machines all our human knowledge and train them to take appropriate courses of action. Learning about the world and figuring out how to act is the foundation of human behavior and consequently, machine learning is the building block of AI.

An Approach to Achieve AI by Teaching Machines to Learn

On a quest for artificial intelligence, Arthur Samuel at IBM coined the term "Machine Learning" in 1959. Machine Learning involves a lot of math as you'll see in a little bit but it's only a narrow section of data science. This field is also very similar to Statistics and that's because it's Applied Statistics.

Believe it or not, "intelligent" human decisions and actions can be boiled down to just mere patterns and can be recreated utilizing math. Remember how you first learned how to cross the street? You waited till the lights turned and the cars on the left and right of you stopped at their appropriate lines. If a car suddenly started moving, you reassessed if it was safe to cross the road. Do you recognize the patterns? If the light is red, stop. Else if the light is not red, walk. How our decisions involve math are a little more subtle. If we detect a car moving towards us when we're walking, we predict the possibility that we need to stop walking or move away. The closer the car's angle is towards us and the faster the speed it's going, the higher the percentage we need to run.

While humans learn from experiences, machines learn from " data" (when we tag ourselves in our Facebook photos, our text messages, and any sets of data we supply it.) The more experiences humans have and the more data the machine has, the more accurate we are/it is. Three ways we teach machines are Supervised, Unsupervised, and Reinforcement Learning. There is a long list of algorithms to accomplish each type of learning. We'll only look at some. " Algorithms" in machine learning context are sets of step by step mathematical instructions we give the machine to run. After a machine runs through this algorithm, it becomes a " model " (a hypothesis of how it thinks the world works.) These models undergo constant improvement so the machine perfects the skill. Let's look at examples.

Is This a Cat?

One way humans learn is through " training." We can classify a cat because we've been told what a cat is and we've seen more than enough cats to count. We can classify if an email is spam because we know what spam mail is like. We can infer we were just served wine because of the color of the liquid and the alcohol content. We can predict the cost of a house by looking at its location, size, and amenities. We can predict if a cancer is malignant by looking at patterns of previous cancers. To train a machine to learn this way, we give it a set of " training data" with " desired output."

To "train" the machine to classify objects like cats, we find millions of cat photos on the internet. The more data we give the machine, the more accurate. So we include pictures of cats of different colors, different sizes, at different angles, yawning, jumping, stretching, and anything else we can think of. We give the machine the "desired output" for this training dataset by labeling the photos with "cat." After the machine has gone through a popular algorithm called the "Classification algorithm," it has learned a "model" of how the world of cats work.

Now to test the model and see if it learned, we give it a photo and ask if it's a cat. The machine begins to run the model. Looking at each " feature" in the photo, it'll give it a " weight." After some more calculations we instructed it to do, it'll have computed percentages (e.g. 50 percent chair, 70 percent fluffy rug, 30 percent cat). At the end, it'll return the class with the highest numerical value - fluffy rug. If the model told us it's a cat, we have to train it with even more photos. Maybe more photos of white cats. This is an example of how machines can identify/classify objects.

To train the machine to predict the possibility of a "yes" or "no," we might give it training data of previous spam emails. Let's say 40 percent of the emails in this training data with the phrase "Last Chance" was labeled spam and 85 percent of them with the phrase "Act Now" was labeled spam. After the machine is given this training data, it has a model of how it thinks spam emails work.

Now we test our model with a new email. Within our model, we might instruct it to say yes to any percentage higher than 80 and no to anything lower. When it receives a new email, searches and detects the phrase "Act Now," it will label this email with 85 percent chance of being spam. Since this percentage is over the 80 percent that we set, it will return yes. This is the " Naive Bayes algorithm. "

To teach the machine how to predict and return numbers such as house prices, we might give it training data of existing house prices. It runs through a " Linear Regression algorithm" and arrives at a model that looks like the chart below. Its model hypothesizes a linear correlation between the size of the house in square feet and the price. With this model, it'll confidently predict a number based on where it plots the house.

Unsupervised Learning

Another way humans learn is by discovering patterns. We learn our credit card is stolen when we discover abnormal high spending. We learn that drinking a cold beer or a cold sangria is more satisfying than drinking room temperature red wine on a hot summer day. We learn that if we've watched a few horror movies and still don't like it, we probably don't want to watch any more horror movies. This instance of learning is when we have a lot of experiences and data but instead of having an answer key to match or compare new experiences against, we ask the machine to crunch some numbers to look for and discover patterns that lead to brand new conclusions that we might not have thought of.

To see if there's anything interesting in bank transactions, we might feed a machine customer transaction data. It'll look at the features and compare their consistency for any patterns. If there's a purchase that's a significant outlier (spending is abnormally high or abnormally low compared to the rest of the data), the machine just discovered potential credit card fraud. Now you see how Chase Bank probably figured out how to email you to confirm a suspicious transaction.

Association Discovery

When we feed a machine data on our sleep patterns, it might learn to associate variables. It might associate human sleep hours to usually be around 11 p.m. to 5 a.m. Of course, the more data we give it, the more accurate (for those who argue they never sleep!) For now the machine has learned that a great time to plot human destruction is between the hours 11 p.m. to 5 a.m. Just kidding.

To find meaningful data in the entertainment world, we might feed a machine data on customer habits. It might notice that there's a high frequency of customers buying baby diapers and baby cribs. The machine might not know what diapers or cribs are but based on the calculations of how many times they were brought together versus how many times diapers and bananas were bought together, it knows there's something going on. It now starts to group similar products together. This is the same for movies and video watching. Now you get a glimpse of how Amazon, Youtube, and Netflix's "recommendation system" works.

Reinforcement Learning

The last way we learn is by discovering steps and strategies and training ourselves over and over so that we can have a higher chance of getting what we want.

Games are a classic example. Think of a game where you've played over and over until you've learned the sequence of steps you need to take to win. Now, imagine playing out all the possibilities and being able to store all the steps that led you to win in your head. Next time you play, you know exactly what moves you should make no matter what move your opponent makes. Or do you? It's hard for humans to store and recall all this memory but machines can.

Understanding this, it's no surprise machines have successfully beat humans in games. IBM's Deep Blue machine beat world class chess champion Garry Kasparov in chess in 1997 and Google's AlphaGobeat world class champion Lee Sedol in Go in 2016 (supposedly said to be the hardest game.)

Tic-tac-toe is a fairly simple game where each player is either "X" or "O" and takes turns placing a piece on an empty square on a board until three consecutive pieces are found. Machine learning experts call the player " agent," the end goal of winning with three consecutive pieces the " reward," the Tic-tac-toe game as the " environment," the current game state as the " state," and the move taken as the " action." The important thing to note is how the machine learns what to do in a complex uncertain environment with no idea how its actions will affect it attaining its reward. Like humans, it can only sense its environment by reassessing how every action got it closer or farther from its reward.

Every time humans play, we pick up strategies (e.g. starting first has an advantage, or putting your piece in the center limits your opponent's chances of winning.) When the machines pick up these strategies, they are more mathematically accurate.

Utilizing math (that I'll make up), they might remember that starting first has a 75 percent chance of winning because out of the 2,000 times it played, it won 1,000 times and out of the 1000 times, 750 times it had the first move. Every step it makes, we tell it to crunch numbers and associate them with different variables. To decide which box to put its game piece next, we've instructed it to crunch some more numbers to figure out which box will increase the number associated with the reward. Every time after the machine makes a move, it reassesses its state and recalculates its reward. If its action led it closer to winning, the number associated with its reward increases, and vice versa.

We instruct the machines to basically repeat these mathematical steps over and over again; storing data of its moves, recalculating its chances of winning with every move, updating the number associated with its reward, and deciding on the next best possible move by figuring out what move would minimize loss of the reward. We might instruct the machine to keep playing against different versions of itself over and over and maybe only stop when it wins 100 percent of the last 100 games it played. This is the "Minimax algorithm."

Deep Learning rose out as a subfield to Machine Learning. While Machine Learning is a way to achieve AI, Deep Learning is a way to implement Machine Learning given data of many dimensions. Similar to the biology of our human brains, we put data, " input", through multiple " neural networks," in " neural layers," and get " output." Each function (task) is a " neuron" and each neuron is a " layer." The more networks, the more complex. Hence the name "Deep" Learning. Facial recognition (recognizing whose face is seen), computer vision (classifying and recognizing what's happening in an image), and speech recognition (translating your spoken speech to machine understandable text) use deep learning. Let's look at examples.

A Neural Network

Text Recognition

Deep learning is a subset of many neural networks. But what is a single neural network? How do you think a computer might recognize a "9" written with a sloppy handwriting? Let's look at possible layers the machine might go through until it reaches the conclusion that the digit "9" is written.

The "9" is represented within a 28 by 28-pixel canvas. One layer (neuron) might be to recognize the parts of the text that form a round circle or the parts that form lines. There might be special " weights " (a special numerical value) given to special features meaning we value that feature the most (e.g. the round circle). This neuron now holds a number and affects the next neuron. The second layer might be to put together that one circle is found and one straight line is found.

We might add a " bias " factor because it might be easy to find straight lines and that's biased, so some more math is crunched. The last layer might go through all the numerical digits to see which is composed of one circle and one line. Imagine repeating these steps for a sentence, in the forms of numbers, alphabets, and special characters ("?", ".", "{}", etc.)

As with everything in machine learning, the models need to be tested and refined. If the "9" was not recognized, we would tune the parameters we give it such as the weight and the bias factors. In order to accurately and delicately not completely flip the output, experts will use a " sigmoid function " that uses math derivatives to tune the parameters. We always perfect the models.

This approach can be used for a number of tasks. One would be face recognition. To detect if a human face exists or not, it takes many machine learning algorithms to put together and recognize a face. So one neural layer in this deep learning process might be classifying and identifying specific features (teeth, pupil, nostrils.) Another layer might determine generalizations (if it's a mouth, an eye, a nose.) Another layer might determine if this is, in fact, a human face or not.

Voice recognition is another task. Between the time you speak and the time a voice assistant like Alexa responds back to you, think about the many dimensions of data you are passing it. Multiple neural networks are assessing the input it's receiving: which words you are saying, the language you're speaking, if you're asking a question.

How to Perfect Each ML, DL Model

Now that you've seen the three types of machine learning (supervised, unsupervised, and reinforced) and deep learning, there are three steps to learning. Based on what we know from our experiences/data, we can " predict/infer" something. Every time we predict/infer wrong, we learn from our " error/loss." After each experience, we " refine/retrain" ourselves. This is the same way we perfect our machines after they train on these algorithms (set of step by step mathematical instructions) and create a model (hypothesis about how the world works.) After the models predict/infer, their answers go into another set of instructions that checks if it made an error. Whenever it does, we refine and tune our " hyperparameters" (the kinds of data like cat photos that we supply it) until we are satisfied with the model's performance.

How to Know Which Model to Choose for Each Task

There are many algorithms that can answer the same question or accomplish the same task. How do we know which to choose? We can predict how much a house will cost using unsupervised learning's association discovery algorithm or supervised learning's linear regression algorithm. In order to find the best algorithm, we put these models to the test by collecting the results they give and then seeing how " precise" they are each time and how " often they fail." May the best algorithm win.

That was a lot! But hopefully, you've seen the tip of the iceberg of what machine learning is and how the combination of machine learning approaches into neural networks and deep learning can empower Artificial Intelligence with human intelligence and decision making. After training and refining the models, we essentially get " predictive models" that can help answer questions to things we should know from our data (e.g. Is this cancer malignant?) or help us predict unknowns (e.g. What trends are there in human behavior?) To contribute to machine learning and/or deep learning, we'd just need to sharpen our math skills, learn some programming languages, learn some frameworks, and be on our way to contributing to AI.

# data[].attributes.attachment_link.title
Machine Learning Made Easy - The New Stack

# data[].attributes.attachment_link.description
Remember the movie "The Imitation Game"? The tragic story of a brilliant man who decrypted secret German Enigma messages, indirectly shortened World War II, saved millions of lives, and was later charged for homosexuality, forced to undergo chemical treatment, and ended his life shortly after?

# data[].attributes.attachment_link.title
A Beginner's Guide to Understanding Python Lambda Functions

# data[].attributes.attachment_link.description
Lambdas in Python are one of the most useful, important, and interesting features to know about. Here's how to use them and why they're useful.

# data[].attributes.attachment_link.readable_text
Artificial Intelligence/Machine Learning field is getting a lot of attention right now, and knowing where to start can be a little difficult.

I've been dabbling in this field, so I thought of curating the best resources in one place. All of these are curated based on if it's an inspiring read or a valuable resource. I hope this curated list help you get started on what you need to know about AI/Machine Learning on a technical level.

Learning From Data - Online Course (MOOC) -   More than 4M views on Youtube and iTunes. Featured on edX. Free Course, Lecture Videos Available.

Machine Learning - Free Course with Lecture Slides and Video recordings
Department of Computer Science, 2014-2015, ml, Machine Learning.

Neural Networks for Machine Learning - Learn about artificial neural networks and how they're being applied to speech and object recognition, image segmentation, modeling language and human motion. Free Course.

Data Science Weekly Newsletter -  A free weekly newsletter featuring curated news, articles and jobs related to DataScience.

Did I miss any awesome blogs/videos/resources? Let me know on Twitter or comments.

Found this post useful? Kindly tap the ❤ button below! :)

This list was originally curated on Nucleus. You can check out more awesome machine learning resources here.

# data[].attributes.attachment_link.title
Index of Best AI/Machine Learning Resources - Hacker Noon

# data[].attributes.attachment_link.description
Artificial Intelligence/Machine Learning field is getting a lot of attention right now, and knowing where to start can be a little difficult. I've been dabbling in this field, so I thought of curating the best resources in one place. All of these are curated based on if it's an inspiring read or a valuable resource.

# data[].attributes.attachment_link.readable_text
The blockchain seems to be all that anyone is talking about these days in the world of investing. Bitcoin this, Ethereum that, but what exactly is the blockchain? Is it literally a connected line of squares and cubes? Nope. Is it an online bank where people cash in and cash out their digital crypto tokens? Nope again.

Very simply, it is an online ledger that is connected to various nodes or computers. It is a massive digital record of crypto transactions. While people normally say "the blockchain" or place the word "technology" on the end of the term, they mean different things. Not only that, there's actually more than one type. Every different cryptocurrency has its own.

Not every blockchain is the same or performs the same functions. But a good rule of thumb is that they generally involve data storage of financial transactions, they update in real time, they often use digitally encrypted signatures and contracts to ensure privacy between transactors, and more. You don't need to learn all the ins and outs of the technology to invest in cryptos, but hopefully, this gives you a good grasp of what this technology is and what it does. It's definitely becoming more and more important and relevant info in modern society, so do pay
attention.
The Three Pillars
What are the "three pillars" you may have heard whispered around parts of the internet lately? That's simply a phrase used to refer to the idea that a good blockchain has three fundamental characteristics that it always needs to prioritize and keep strong: decentralization, security, and scalability.
Decentralization
You've probably heard this word a lot in reference to Bitcoin and cryptocurrencies in general. This is fair considering it's arguably the main selling point of crypto-technology and blockchain technology in the first place. Decentralization refers to any kind of technology that is not monopolized or owned by anyone settled institution or organization. Think of it like open-source software combined with the benefits of being P2P or peer to peer.

Centralization implies there is a central server that all interacting users are actually sending their info to. The central hub is like a large node that is connected to all of the other nodes. But decentralization strips away the use of any sort of central hub and only leaves behind a network of independent, individual users. It is this collection of users that make the very foundation of the blockchain itself, not a central node or hub.

Since these digital currencies are not tied to any one particular bank such as the USD or any fiat currency for that matter, they are traded on various cryptocurrency exchanges. It's essential to choose one that is secure and trustworthy if you're going to invest in and trade cryptos, which brings us to the next pillar.
Security
This pillar is a no-brainer. One of the biggest concerns of anyone who delves into the world of cryptocurrency is how safe it is. Something with no safety protocols for its users or is prone to hacking attacks, is fundamentally useless.

Security in this sort of framework generally means safety and privacy. As a matter of fact, privacy also goes hand in hand with decentralization. People want their assets to be kept to themselves and, if or whenever possible, only themselves.

There are a variety of ways that people ensure they have secure access to their funds or assets. This is often done through crypto wallets, which are a great example of this technology at work. Wallets have their own digital address and can be part of online service platforms that connect you to a specific blockchain and allow you to buy and sell. The funds are automatically taken out of or put into your storage online based off of your own security and verification settings that the site allows you to customize.

There are also hardware wallets which are like little keychains you plug into your computer so that you can transfer your funds from an online database to your very own hands in the real
world.
Scalability
This last pillar is the least talked about of the three but has been seeing some buzz and hype lately concerning the future of blockchain technology. Scalability simply refers to a blockchain's ability to grow in size while maintaining smooth operations. "Smooth operations" often means keeping the network free of slow processing times, system bloating, lag, but especially high transfer rates.

As more and more people begin trading in Bitcoin, for example, the Bitcoin blockchain gets larger and larger. The issue this brings up is how well equipped the Bitcoin blockchain can handle millions upon millions of nodes and users before it becomes too slow to remain a viable system.

This is why new blockchain services are popping up lately with the main intent of creating new infrastructure to increase efficiency and performance. This is often measured by transactions per second. Scalability, then, refers to a blockchain's ability to grow and adapt to the larger number of users and higher volume of data.
Conclusion
While blockchains are incredibly complicated and hard-to-measure systems that literally get more complex by the minute, this short but concise guide has covered the basics.

-

This Article is Brought to you By Our Site Sponsor: John Michelson

Photo Credit: Provided By the Author

# data[].attributes.attachment_link.title
Blockchain for the Clueless: A Brief Guide to the Three Pillars of Decentralization - The Good Men Project

# data[].attributes.attachment_link.description
The blockchain seems to be all that anyone is talking about these days in the world of investing. Bitcoin this, Ethereum that, but what exactly is the blockchain? Is it literally a connected line of squares and cubes? Nope. Is it an online bank where people cash in and cash out their digital crypto tokens?

# data[].attributes.attachment_link.readable_text
Introduction

From Facebook's research to DeepMind's legendary algorithms, deep learning has climbed its way to the top of the data science world. It has led to amazing innovations, incredible breakthroughs, and we are only just getting started!

However if you are a newcomer to this field, the word "deep" might throw you into doubt. Deep learning is one of the hottest topics of this industry today, but it is unfortunately foreign and cryptic to most people. A lot of people carry an impression that deep learning involves a lot of mathematics and statistical knowledge.

If you had similar questions about deep learning, but were not sure how, when and where to ask them - you are at the right place. This article should answer most of what you would want to know.

By end of this article, we will dispel a few myths about deep learning and answer some widely asked questions about this field. We have also included plenty of resources to get you started.

Here is the exciting part - It isn't as difficult as most people make it out to be. Read on to find more!

Table of Contents

What is Deep Learning and why is it so popular?

Is Deep Learning just a hype or does it have real-life applications?

What is the difference between Deep Learning and Machine Learning?

What are the prerequisites for starting out in Deep Learning?

Is it necessary to have a research background in Deep Learning to start out in it?

Which Tools/Languages should I prefer to build Deep Learning models?

Why are GPUs necessary for building Deep Learning models?

When (and where) to apply Deep Learning?

Do we need a lot of data to train deep learning models?

Where can I find some basic project ideas in order to practice deep learning?

What are the free learning resources for Deep Learning?

What are Deep Learning interview questions?

What is the future of Deep Learning?

What is Deep Learning and why is it so popular these days?

Deep Learning is nothing but a paradigm of machine learning which has shown incredible promise in the recent years. This is because of the fact that Deep Learning shows great analogy with the functioning of the human brain. The superiority of the human brain is an evident fact, and it is considered to be the most versatile and efficient self-learning model that has ever been created.

Let us understand the functioning of a deep learning model with an example:
What do you see in the above image?
The most obvious answer would be " ", right? Despite the fact, that there is sand, greenery, clouds and a lot of other things, our brain tags this image as one of a car. This is because our brain has learnt to identify the primary subject of an image.

This ability of deriving useful information from a lot of extraneous data is what makes deep learning special. With the amount of data that is being generated these days, we want our models to be better with more of this data being fed into it. While deep learning models get better with the increase in the amount of data.

Now although Deep Learning has been around for many years, the major breakthroughs from these techniques came just in the recent years. This is because of two main reasons - the first and foremost, as we saw before, is the increase of data generated through various sources. The infographic below succinctly visualizes this trend. The second is the growth in hardware resources required to run these models. GPUs, which are becoming a requirement to run deep learning models, are multiple times faster and they help us build bigger and deeper deep learning models in comparatively less time than we required previously.

This is the reason that Deep Learning has become a major buzz word in the data science industry.

Is Deep Learning just a hype or does it have real-life applications?

Deep Learning has found many practical applications in the recent past. From Netflix's famous movie recommendation system to Google's self-driving cars, deep learning is already transforming a lot of businesses and is expected to bring about a revolution in almost all industries. Deep learning models are being used from diagnosing cancer to winning presidential elections , from creating art and writing literature to making real life money. Thus it would be wrong to say that it is just a hyped topic anymore.

Some major applications of deep learning that are being employed by technology companies are:

Google and Facebook are translating text into hundreds of languages at a time. This is being done through some deep learning models being applied to NLP tasks and is a major success story.

Conversational agents like Siri, Alexa, Cortana basically work on simplifying the speech recognition techniques through LSTMs and RNNs. Voice commands have added a whole new domain to the possibilities of a machine.

Deep learning is being used in impactful computer vision applications such as OCR (Optical Character Recognition) and real time language translation

Multimedia sharing apps like Snapchat and Instagram apply facial feature detection which is another application of deep learning.

Deep Learning is being used in Healthcare domain to locate malignant cells and other foreign bodies in order to detect complex diseases.

However, some people develop a thinking that deep learning is overhyped because of the fact that labeled data required for training deep learning models is not readily available. Even if the data is available, the computational power required to train such models does not come cheap. Hence, due to these barriers, people are not able to experience the power of deep learning and term it as just hype.

Go through the following blog to build some real life deep learning applications yourself:

What is the difference between Deep Learning and Machine Learning?

This is one of the most important questions that most of us need to understand. The comparison can be done mainly on the below three verticals:
Data dependencies
The most important difference between deep learning and traditional machine learning is its performance as the scale of data increases. When the data is small, deep learning algorithms don't perform that well. This is because deep learning algorithms need a large amount of data to understand it perfectly. On the other hand, traditional machine learning algorithms with their handcrafted rules prevail in this scenario. Below image summarizes this fact.
Feature engineering
Feature engineering is a process of putting domain knowledge into the creation of feature extractors to reduce the complexity of the data and make patterns more visible to learning algorithms to work. This process is difficult and expensive in terms of time and expertise.

In Machine learning, most of the applied features need to be identified by an expert and then hand-coded as per the domain and data type.

For example, features can be pixel values, shape, textures, position and orientation. The performance of most of the Machine Learning algorithm depends on how accurately the features are identified and extracted.

Deep learning algorithms try to learn high-level features from data. This is a very distinctive part of Deep Learning and a major step ahead of traditional Machine Learning. Therefore, deep learning reduces the task of developing new feature extractor for every problem. Like, Convolutional NN will try to learn low-level features such as edges and lines in early layers then parts of faces of people and then high-level representation of a face.
Interpretability
Last but not the least, we have interpretability as a factor for comparison of machine learning and deep learning. This factor is the main reason deep learning is still thought 10 times before its use in industry.

Let's take an example. Suppose we use deep learning to give automated scoring to essays. The performance it gives in scoring is quite excellent and is near human performance. But there's is an issue. It does not reveal why it has given that score. Indeed mathematically you can find out which nodes of a deep neural network were activated, but we don't know what there neurons were supposed to model and what these layers of neurons were doing collectively. So we fail to interpret the results.

On the other hand, machine learning algorithms like decision trees give us crisp rules as to why it chose what it chose, so it is particularly easy to interpret the reasoning behind it. Therefore, algorithms like decision trees and linear/logistic regression are primarily used in industry for interpretability.

If you would like to learn about a more in-depth comparison between machine learning and deep learning, I recommend you go through the following blog:

What are the prerequisites for starting out in Deep Learning?

Starting out in deep learning is not as difficult as people might make you believe. There are a few elementary basics that you should cover before diving into deep learning. Deep learning requires knowledge of the following topics:

Mathematics: You should be comfortable with probability, derivatives, linear algebra and a few other basic topics. Khan Academy offers a decent course covering almost all the above topics .

Statistics: The basics of statistics are required for going forward with any machine learning problem. Understanding the concepts of statistics are essential because most of the deep learning concepts are derived from assimilating the concepts of statistics. You can check the online courses available here .

Tool: A decent level of coding skills are required for implementing deep learning into real life problems. Coursera's, Introduction to Data Science in Python is a decent course to start off with Python as a tool.

Machine Learning: Machine learning is the base for deep learning. One can not start learning deep learning without understanding the concepts of machine learning. You could go through Intro to Machine Learning or Andrew Ng's course Machine Learning for a theoretical base.

For a more detailed understanding about the prerequisites please follow:

Do I need to do a PhD to make a career in Deep Learning?

No, a PhD is not a mandatory requirement to make a career in deep learning. You can learn, experiment, and build up your work experience portfolio without going to university. The emphasis for any job or role is usually on demonstrating your competence, and not on your degree, per se.

Having said that, a PhD in a specific field (like linguistics for NLP) will definitely accelerate your path if you choose to combine that with deep learning.

Which Tools/Languages should I prefer to build Deep learning models?

I would recommend you use Python, because of its robust ecosystem for machine learning. The python ecosystem comprises of developers and coders who are providing open source libraries and support for the community of python users. This makes the task of writing complex codes for various algorithms much easier and the techniques easier to implement and experiment with.

Also, Python being a more generalized programming language, can be used for both the development and implementation. This greatly simplifies the transition from development to operations. That is, a deep learning product that can predict the price of flight tickets, can not only be developed in python but can also be attached with your website in the same form. This is what makes Python a universal language.

Besides this, I would suggest that beginner's use high level libraries like Keras. This makes experimentation easier by providing abstraction to the unnecessary information that is hidden under the algorithms. And giving access to the parameters that can be tweaked to enhance the performance of such models. Let us understand this with an example:

When you press the buttons on a television remote, do you need to care about the background processes that are happening inside the remote? Do you need to know about what signal is being sent out for that key, or how is it being amplified?

Because maybe an understanding of these processes is required for a physicist but for a lame man sitting in his bedroom, it is just an information overload.

There are also other contenders apart from Python in the deep learning space such as R, Julia, C++, and Java. For alternatives of libraries, you can check out TensorFlow, Pytorch, Caffe2, DL4J, etc. We should stay updated with their developments as well.

If you are not well versed with programming, there are also a few GUI based softwares, that require no coding, to build deep learning models, such as Lobe or Google's AutoML, among others.

Why are GPUs necessary for building Deep Learning models?

When you train a deep learning model, two main operations are performed:

In forward pass, input is passed through the neural network and after processing the input, an output is generated. Whereas in backward pass, we update the weights of neural network on the basis of error we get in forward pass.

Both of these operations are essentially matrix multiplications. A simple matrix multiplication can be represented by the image below

Here, we can see that each element in one row of first array is multiplied with one column of second array. So in a neural network, we can consider first array as input to the neural network, and the second array can be considered as weights of the network.

This seems to be a simple task. Now just to give you a sense of what kind of scale deep learning - VGG16 (a convolutional neural network of 16 hidden layers which is frequently used in deep learning applications) has ~140 million parameters; aka weights and biases. Now think of all the matrix multiplications you would have to do to pass just one input to this network! It would take years to train this kind of systems if we take traditional approaches.

We saw that the computationally intensive part of neural network is made up of multiple matrix multiplications. So how can we make it faster?

We can simply do this by performing all the operations at the same time instead of doing it one after the other. This, in a nutshell, is why we use GPU (graphics processing units) instead of a CPU (central processing unit) for training a neural network.

When (and where) to apply Neural Networks ?

Deep Learning have been in the spotlight for quite some time now. Its "deeper" versions are making tremendous breakthroughs in many fields such as image recognition, speech and natural language processing etc.

Now that we know it is so impactful; the main question that arises is when to and when not to apply neural networks? This field is like a gold mine right now, with many discoveries uncovered everyday. And to be a part of this "gold rush", you have to keep a few things in mind:

Firstly, deep learning models require clear and informative data (and mostly big data) to train. Try to imagine deep learning model as a child. It first observes how its parent walks. Then it tries to walk on its own, and with its every step, the child learns how to perform a particular task. It may fall a few times, but after few unsuccessful attempts, it learns how to walk. If you don't let it, it might not ever learn how to walk. The more exposure you can provide to the child, the better it is.

It is prudent to use Deep Learning for complex problems such as image processing. Deep Learning algorithms belong to a class of algorithms called representation learning algorithms. These algorithms break down complex problems into simpler form so that they become understandable (or "representable"). Think of it as chewing food before you gulp. This would be harder for traditional (non-representation learning) algorithms.

When you have an appropriate type of deep learning to solve the problem. Each problem has its own twists. So the data decides the way you solve the problem. For example, if the problem is of sequence generation, recurrent neural networks are more suitable. Whereas, if it is image related problem, you would probably be better of taking convolutional neural networks for a change.

Last but not the least, hardware requirements are essential for running a deep neural network model. Neural nets were "discovered" long ago, but they are shining in the recent years for the main reason that computational resources are better and more powerful. If you want to solve a real life problem with these networks, get ready to buy some high-end hardware!

Do we need a lot of data to train deep learning models?

It is true that we need a large amount of data to train a typical deep learning model. But we can generally overcome this by using something called transfer learning. Let me explain thoroughly.

One of the barrier for using deep learning models for industry applications is where the data is not in huge amount. A few examples of data needed to train some of the popular deep learning models are:

However, a deep learning model trained on a specific task can be reused for different problem in the same domain even if the amount of data is not that huge. This technique is known as Transfer Learning.

For instance, we have a set of 1000 images of cats and dogs labeled as 1 and 0 (1 for cat and 0 for dog) and we have another set of 500 test images that we need to classify. So, instead of training a deep learning model on the data of 1000 images, we can use a pre-trained VGGNet model and retrain it on our data and use it to classify the unlabeled set of images. A pre-trained model may not be 100% accurate in your application, but it saves huge efforts required to reinvent the wheel.

You may have a look at this to get a better intuition of using a pre-trained model.

Where can I find basic project ideas in order to practice deep learning?

To practice deep learning, ideas alone will not help. We also need labeled data to test our ideas using deep learning.

1. For beginners, getting started with the is highly recommended. The dataset contains handwritten digits with their actual labels, i.e., numbers from 0 to 9. You can even compete in the Identify the Digits competition to evaluate your models

2. For intermediate users, this Age Detection challenge is a nice project to work on. The dataset consists of facial images of Indian movie actors. The task is to predict the age of a person from his or her facial attributes. For simplicity, the problem has been converted to a multiclass problem with classes as Young, Middle and Old.

You can also refer this of exciting deep learning datasets and problems.

What are some of the free learning resources for Deep Learning?

Being a comparatively newer technology, there is not enough content and tutorials available for the beginners. However, free-quality content and resources related to deep learning are steadily increasing. The learning resources can be classified on the different applications of deep learning.

1. Convolutional Neural Networks by Andrew Ng

2. Architecture of Convolutional Neural Networks (CNNs) demystified

3. A Beginner's Guide To Understanding Convolutional Neural Networks

4. Machine Learning is Fun! Part 3: Deep Learning and Convolutional Neural Networks

5. ImageNet Classification with Deep Convolutional Neural Networks

6. Going Deeper with Convolutions

Sequence Prediction/Time Series:

1. Sequence Models by Andrew Ng

2. Essentials of Deep Learning : Introduction to Long Short Term Memory

3. The Unreasonable Effectiveness of Recurrent Neural Networks

4. Understanding LSTM Networks

5. Recurrent Neural Networks & LSTMs

Natural Language Processing:

1. Deep Learning for Natural Language Processing

2. Distributed Representations of Words and Phrases and their Compositionality

3. Sequence to Sequence Learning with Neural Networks

4. Deep Speech: Scaling up end-to-end speech recognition

Besides this, you can also go through the following blogs for a more extensive list of resources:

What are some Deep Learning interview questions?

Some common questions that may be asked on deep learning are:

1. How do deep learning models learn?

2. What are some limitations of a deep learning model?

3. What are the differences between feedforward neural networks and recurrent neural networks?

4. What are activation functions and why are they required?

5. What is a CNN and what are its applications?

6. What is pooling? How does it work?

7. What is a dropout layer and why is it used?

8. What is the vanishing gradient problem and how do we overcome that ?

9. What are optimization functions? Name a few of the common optimization functions.

Do note that this is not an exhaustive list that will make you completely ready for an interview. You can go through the following skill test to test yourself on important questions on deep learning.

What is the future of Deep learning?

Deep learning has come a long way in recent years, but still has a lot of untapped potential. We are still in the nascent stages of this field, with new breakthroughs happening seemingly every day. One of the use-cases that we can definitely see in the suture is of automobile industry, where Deep Learning can revolutionize it by making self-driving cars a reality. While we don't have a crystal ball to predict the future, we can see deep learning models requiring less and less involvement from human data scientists and researchers.

In the immediate future, we can definitely see a trend where the knowledge of deep learning will be a skill required by every Data Science practitioner. In fact, you must have caught sight of a job position spurn out recently, called a "Deep Learning Engineer". This person is responsible to deploy and maintain Deep Learning models used by various departments of that company. Needless to say, there will be a huge demand of such people in the industry.

Currently, one of the limitations of DL is that it does what a human asks of it. It requires tons of data to learn it's target objective, and replicates that. This has induced bias in certain applications. We can see this improving over time such that the bias is eliminated in the training process.

We might even stop differentiating deep learning from the other types of learning, with time. It is primed to become a popular and commonly used field and will not require special branding efforts to market or sell.

There are a lot of cases still where researchers, after training a DL model, are unable to explain the 'why' behind it. "It's producing great results but why did you tune a hyperparameter a certain way?" Hopefully with the rapid advancement in DL, we will see this black box concept becoming history, and we can explain the intuition behind the decision it takes.

End Notes

These are broadly the answers to the most frequently asked questions on our portal or elsewhere by the people who want to jump onto this Deep Learning bandwagon.

Do you have any other questions on deep learning that you need clarification on? Use the comments section below to ask; or hop onto our Discussion portal, and we will help you out!

You can also read this article on Analytics Vidhya's Android APP

# data[].attributes.attachment_link.title
12 Frequently Asked Questions on Deep Learning (with their answers)!

# data[].attributes.attachment_link.description
Introduction From Facebook's research to DeepMind's legendary algorithms, deep learning has climbed its way to the top of the data science world. It has led to amazing innovations, incredible breakthroughs, and we are only just getting started! However if you are a newcomer to this field, the word

# data[].attributes.attachment_link.readable_text
Originally published at https://www.datacamp.com/community/tutorials/tensorflow-tutorial
Deep learning is a subfield of machine learning that is a set of algorithms that is inspired by the structure and function of the brain.

TensorFlow is the second machine learning framework that Google created and used to design, build, and train deep learning models.You can use the TensorFlow library do to numerical computations, which in itself doesn't seem all too special, but these computations are done with data flow graphs. In these graphs, nodes represent mathematical operations, while the edges represent the data, which usually are multidimensional data arrays or tensors, that are communicated between these edges.

You see? The name "TensorFlow" is derived from the operations which neural networks perform on multidimensional data arrays or tensors! It's literally a flow of tensors. For now, this is all you need to know about tensors, but you'll go deeper into this in the next sections!

You'll first learn more about tensors;

Then, the tutorial you'll briefly go over some of the ways that you can install TensorFlow on your system so that you're able to get started and load data in your workspace;

After this, you'll go over some of the TensorFlow basics: you'll see how you can easily get started with simple computations.

After this, you get started on the real work: you'll load in data on Belgian traffic signs and exploring it with simple statistics and plotting.

In your exploration, you'll see that there is a need to manipulate your data in such a way that you can feed it to your model. That's why you'll take the time to rescale your images and convert them to grayscale.

Next, you can finally get started on your neural network model! You'll build up your model layer per layer;

Once the architecture is set up, you can use it to train your model interactively and to eventually also evaluate it by feeding some test data to it.

Lastly, you'll get some pointers for further improvements that you can do to the model you just constructed and how you can continue your learning with TensorFlow.

Today's TensorFlow tutorial for beginners will introduce you to performing deep learning in an interactive way:

Download the notebook of this tutorial here.

Also, you could be interested in a course on Deep Learning in Python, DataCamp's Keras tutorial or the keras with R tutorial.

Introducing Tensors

To understand tensors well, it's good to have some working knowledge of linear algebra and vector calculus. You already read in the introduction that tensors are implemented in TensorFlow as multidimensional data arrays, but some more introduction is maybe needed in order to completely grasp tensors and their use in machine learning.

Before you go into plane vectors, it's a good idea to shortly revise the concept of "vectors"; Vectors are special types of matrices, which are rectangular arrays of numbers. Because vectors are ordered collections of numbers, they are often seen as column matrices: they have just one column and a certain number of rows. In other terms, you could also consider vectors as scalar magnitudes that have been given a direction.

Remember: an example of a scalar is "5 meters" or "60 m/sec", while a vector is, for example, "5 metres north" or "60 m/sec East". The difference between these two is obviously that the vector has a direction. Nevertheless, these examples that you have seen up until now might seem far off from the vectors that you might encounter when you're working with machine learning problems. This is normal; The length of a mathematical vector is a pure number: it is absolute. The direction, on the other hand, is relative: it is measured relative to some reference direction and has units of radians or degrees. You usually assume that the direction is positive and in counterclockwise rotation from the reference direction.

Visually, of course, you represent vectors as arrows, as you can see in the picture above. This means that you can consider vectors also as arrows that have direction and length. The direction is indicated by the arrow's head, while the length is indicated by the length of the arrow.

So what about plane vectors then?

Plane vectors are the simplest setup of tensors. They are much like regular vectors as you have seen above, with the sole difference that they find themselves in a vector space. To understand this better, let's start off with an example: you have a vector that is 2 X 1. This means that the vector belongs to the set of real numbers that come paired two at a time. Or, stated differently, they are part of two-space. In such cases, you can represent vectors on the coordinate (x,y) plane with arrows or rays.

Working from this coordinate plane in a standard position where vectors have their endpoint at the origin (0,0), you can derive the x coordinate by looking at the first row of the vector, while you'll find the y coordinate in the second row. Of course, this standard position doesn't always need to be maintained: vectors can move parallel to themselves in the plane without experiencing changes.

Note that similarly, for vectors that are of size 3 X 1, you talk about the three-space. You can represent the vector as a three-dimensional figure with arrows pointing to positions in the vectors pace: they are drawn on the standard x, y and z axes.

It's nice to have these vectors and to represent them on the coordinate plane, but in essence, you have these vectors so that you can perform operations on them and one thing that can help you in doing this is by expressing your vectors as bases or unit vectors.

Unit vectors are vectors with a magnitude of one. You'll often recognize the unit vector by a lowercase letter with a circumflex, or "hat". Unit vectors will come in convenient if you want to express a 2-D or 3-D vector as a sum of two or three orthogonal components, such as the x− and y−axes, or the z−axis.

And when you are talking about expressing one vector, for example, as sums of components, you'll see that you're talking about component vectors, which are two or more vectors whose sum is that given vector.

Tip: watch this video, which explains what tensors are with the help of simple household objects!

Next to plane vectors, also covectors and linear operators are two other cases that all three together have one thing in common: they are specific cases of tensors. You still remember how a vector was characterized in the previous section as scalar magnitudes that have been given a direction. A tensor, then, is the mathematical representation of a physical entity that may be characterized by magnitude and multiple directions.

And, just like you represent a scalar with a single number and a vector with sequence of three numbers in a 3-dimensional space, for example, a tensor can be represented by an array of 3R numbers in a 3-dimensional space.

The "R" in this notation represents the rank of the tensor: this means that in a 3-dimensional space, a second-rank tensor can be represented by 3 to the power of 2 or 9 numbers. In an N-dimensional space, scalars will still require only one number, while vectors will require N numbers, and tensors will require N^R numbers. This explains why you often hear that scalars are tensors of rank 0: since they have no direction, you can represent them with one number.

With this in mind, it's fairly easy to recognize scalars, vectors, and tensors and to set them apart: scalars can be represented by a single number, vectors by an ordered set of numbers, and tensors by an array of numbers.

What makes tensors so special is the combination of components and basis vectors: basis vectors transform one way between reference frames and the components transform in just such a way as to keep the combination between components and basis vectors the same.

Installing TensorFlow

Now that you know more about TensorFlow, it's time to get started and install the library. Here, it's good to know that TensorFlow provides APIs for Python, C++, Haskell, Java, Go, Rust, and there's also a third-party package for R called tensorflow.

Tip: if you want to know more about deep learning packages in R, consider checking out DataCamp's keras: Deep Learning in R Tutorial.

In this tutorial, you will download a version of TensorFlow that will enable you to write the code for your deep learning project in Python. On the TensorFlow installation webpage, you'll see some of the most common ways and latest instructions to install TensorFlow using virtualenv, pip, Docker and lastly, there are also some of the other ways of installing TensorFlow on your personal computer.

Note You can also install TensorFlow with Conda if you're working on Windows. However, since the installation of TensorFlow is community supported, it's best to check the official installation instructions.

Now that you have gone through the installation process, it's time to double check that you have installed TensorFlow correctly by importing it into your workspace under the alias tf:

import tensorflow as tf

Note that the alias that you used in the line of code above is sort of a convention - It's used to ensure that you remain consistent with other developers that are using TensorFlow in data science projects on the one hand, and with open-source TensorFlow projects on the other hand.

Getting Started With TensorFlow: Basics

You'll generally write TensorFlow programs, which you run as a chunk; This is at first sight kind of contradictory when you're working with Python. However, if you would like, you can also use TensorFlow's Interactive Session, which you can use to work more interactively with the library. This is especially handy when you're used to working with IPython.

For this tutorial, you'll focus on the second option: this will help you to get kickstarted with deep learning in TensorFlow. But before you go any further into this, let's first try out some minor stuff before you start with the heavy lifting.

First, import the tensorflow library under the alias tf, as you have seen in the previous section. Then initialize two variables that are actually constants. Pass an array of four numbers to the constant() function.

Note that you could potentially also pass in an integer, but that more often than not, you'll find yourself working with arrays. As you saw in the introduction, tensors are all about arrays! So make sure that you pass in an array :) Next, you can use multiply() to multiply your two variables. Store the result in the result variable. Lastly, print out the result with the help of the print() function. Find the exercise here.

Note that you have defined constants in the DataCamp Light code chunk above. However, there are two other types of values that you can potentially use, namely placeholders, which are values that are unassigned and that will be initialized by the session when you run it. Like the name already gave away, it's just a placeholder for a tensor that will always be fed when the session is run; There are also Variables, which are values that can change. The constants, as you might have already gathered, are values that don't change.

The result of the lines of code is an abstract tensor in the computation graph. However, contrary to what you might expect, the result doesn't actually get calculated; It just defined the model but no process ran to calculate the result. You can see this in the print-out: there's not really a result that you want to see (namely, 30). This means that TensorFlow has a lazy evaluation!

However, if you do want to see the result, you have to run this code in an interactive session. You can do this in a few ways, as is demonstrated in the DataCamp Light code chunks below.

Note that you can also use the following lines of code to start up an interactive Session, run the result and close the Session automatically again after printing the output.

In the code chunks above you have just defined a default Session, but it's also good to know that you can pass in options as well. You can, for example, specify the config argument and then use the ConfigProto protocol buffer to add configuration options for your session.

For example, if you add config=tf.ConfigProto(log_device_placement=True) to your Session, you make sure that you log the GPU or CPU device that is assigned to an operation. You will then get information which devices are used in the session for each operation. You could use the following configuration session also, for example, when you use soft constraints for the device placement: config=tf.ConfigProto(allow_soft_placement=True).

Now that you've got TensorFlow installed and imported into your workspace and you've gone through the basics of working with this package, it's time to leave this aside for a moment and turn your attention to your data. Just like always, you'll first take your time to explore and understand your data better before you start modeling your neural network.

Belgian Traffic Signs: Background

Even though traffic is a topic that is generally known amongst you all, it doesn't hurt going briefly over the observations that are included in this dataset to see if you understand everything before you start. In essence, in this section, you'll get up to speed with the domain knowledge that you need to have to go further with this tutorial.

Belgian traffic signs are usually in Dutch and French. This is good to know, but for the dataset that you'll be working with, it's not too important!

There are six categories of traffic signs in Belgium: warning signs, priority signs, prohibitory signs, mandatory signs, signs related to parking and standing still on the road and, lastly, designatory signs.

On January 1st, 2017, more than 30,000 traffic signs were removed from Belgian roads.These were all prohibitory signs relating to speed.

Talking about removal, the overwhelming presence of traffic signs has been an ongoing discussion in Belgium (and by extension, the entire European Union).

Of course, because I'm Belgian, I'll make sure you'll also get some anecdotes :)

Loading And Exploring The Data

Now that you have gathered some more background information, it's time to download the dataset here. You should get the two zip files listed next to "BelgiumTS for Classification (cropped images), which are called "BelgiumTSC_Training" and "BelgiumTSC_Testing".

Tip: if you have downloaded the files or will do so after completing this tutorial, take a look at the folder structure of the data that you've downloaded! You'll see that the testing, as well as the training data folders, contain 61 subfolders, which are the 62 types of traffic signs that you'll use for classification in this tutorial. Additionally, you'll find that the files have the file extension .ppm or Portable Pixmap Format. You have downloaded images of the traffic signs!

First, set your ROOT_PATH. This path is the one where you have made the directory with your training and test data.

Next, you can add the specific paths to your ROOT_PATH with the help of the join() function. You store these two specific paths in train_data_directory and test_data_directory.

You see that after, you can call the load_data() function and pass in the train_data_directory to it.

Now, the load_data() function itself starts off by gathering all the subdirectories that are present in the train_data_directory; It does so with the help of list comprehension, which is quite a natural way of constructing lists - it basically says that, if you find something in the train_data_directory, you'll double check whether this is a directory, and if it is one, you'll add it to your list. Remember that each subdirectory represents a label.

Next, you have to loop through the subdirectories. You first initialize two lists, labels and images. Next, you gather the paths of the subdirectories and the file names of the images that are stored in these subdirectories. After, you can collect the data in the two lists with the help of the append() function.

Let's get started with importing the data into your workspace. Let's start with the lines of code that appear below the User-Defined Function (UDF) load_data():

Note that in the above code chunk, the training and test data are located in folders named "Training" and "Testing", which are both subdirectories of another directory "TrafficSigns". On a local machine, this could look something like "/Users/yourName/Downloads/TrafficSigns", with then two subfolders called "Training" and "Testing".

With your data loaded in, it's time for some data inspection! You can start off with a pretty simple analysis with the help of the ndim and size attributes of the images array:

Note that the images and labels variables are lists, so you might need to use np.array() to convert the variables to array in your own workspace. This has been done for you here!

Note that the images[0] that you printed out is in fact one single image that is represented by arrays in arrays! This might seem counterintuitive at first, but it's something that you'll get used to as you go further into working with images in machine learning or deep learning applications.

Next, you can also take a small look at the labels, but you shouldn't see too many surprises at this point.

These numbers already give you some insights into how successful your import was and the exact size of your data. At first sight, everything has been executed the way you expected it to and you see that the size of the array is considerable if you take into account that you're dealing with arrays within arrays.

Tip try adding the following attributes to your arrays to get mor information about the memory layout, the length of one array element in bytes and the total consumed bytes by the array's elements with the flags, itemsize and nbytes attributes. You can test this out in the IPython console in the DataCamp Light chunk above!

Next, you can also take a look at the distribution of the traffic signs.

Awesome job! Now let's take a closer look at the histogram that you made!

You clearly see that not all types of traffic signs are equally represented in the dataset. This is something that you'll deal with later when you're manipulating the data before you start modeling your neural network.

At first sight, you see that there are labels that are more heavily present in the dataset than others: the labels 22, 32, 38, and 61 definitely jump out. At this point, it's nice to keep this in mind, but you'll definitely go further into this in the next section!

The previous, small analyses or checks have already given you some idea of the data that you're working with, but when your data mostly consists of images, the step that you should take to explore your data is by visualizing it.

First, make sure that you import the pyplot module of the matplotlib package under the common alias plt.

Then, you're going to make a list with 4 random numbers. These will be used to select traffic signs from the images array that you have just inspected in the previous section. In this case, you go for 300, 2250, 3650 and 4000.

Next, you'll say that for every element in the lenght of that list, so from 0 to 4, you're going to create subplots without axes (so that they don't go running with all the attention and your focus is solely on the images!). In these subplots, you're going to show a specific image from the images array that is in accordance with the number at the index i. In the first loop, you'll pass 300 to images[], in the second round 2250, and so on. Lastly, you'll adjust the subplots so that there's enough width in between them.

The last thing that remains is to show your plot with the help of the show() function!

Let's check out some random traffic signs:

There you go.

As you sort of guessed by the 62 labels that are included in this dataset, the signs are different from each other.

But what else do you notice? Take another close look at the images below:

These four images are not of the same size!

You can obviously toy around with the numbers that are contained in the traffic_signs list and follow up more thoroughly on this observation, but be as it may, this is an important observation which you will need to take into account when you start working more towards manipulating your data so that you can feed it to the neural network.

Let's confirm the hypothesis of the differing sizes by printing the shape, the minimum and maximum values of the specific images that you have included into the subplots.

The code below heavily resembles the one that you used to create the above plot, but differs in the fact that here, you'll alternate sizes and images instead of plotting just the images next to each other.

Note how you use the format() method on the string "shape: {0}, min: {1}, max: {2}" to fill out the arguments {0}, {1}, and {2} that you defined.

Now that you have seen loose images, you might also want to revisit the histogram that you printed out in the first steps of your data exploration; You can easily do this by plotting an overview of all the 62 classes and one image that belongs to each class.

Note that even though you define 64 subplots, not all of them will show images (as there are only 62 labels!). Note also that again, you don't include any axes to make sure that the readers' attention doesn't dwell far from the main topic: the traffic signs!

As you mostly guessed in the histogram above, there are considerably more traffic signs with labels 22, 32, 38, and 61. This hypothesis is now confirmed in this plot: you see that there are 375 instances with label 22, 316 instances with label 32, 285 instances with label 38 and, lastly, 282 instances with label 61.

One of the most interesting questions that you could ask yourself now is whether there's a connection between all of these instances - maybe all of them are designatory signs?

Let's take a closer look: you see that label 22 and 32 are prohibitory signs, but that labels 38 and 61 are designatory and a prioritory signs, respectively. This means that there's not an immediate connection between these four, except for the fact that half of the signs that has a heavy presence in the dataset is of the prohibitory kind.

Feature Extraction

The size of the images was unequal;

There are 62 labels or target values (as your labels start at 0 and end at 61);

The distribution of the traffic sign values is pretty unequal; There wasn't really any connection between the signs that were heavily present in the dataset.

Now that you have thoroughly explored your data, it's time to get your hands dirty! Let's recap briefly what you discovered to make sure that you don't forget any steps in the manipulation:

Now that you have a clear idea of what you need to improve, you can start with manipulating your data in such a way that it's ready to be fed to the neural network or whichever model you want to feed it to. Let's start first with extracting some features - you'll rescale the images and you'll convert the images that are held in the images array to grayscale. You'll do this color conversion mainly because the color matters less in classification questions like the one you're trying to answer now. For detection, however, the color does play a big part! So in those cases, it's not needed to do that conversion!

To tackle the differing image sizes, you're going to rescale the images; You can easily do this with the help of the skimage or Scikit-Image library, which is a collection of algorithms for image processing.

In this case, the transform module will come in handy, as it offers you a resize() function; You'll see that you make use of list comprehension (again!) to resize each image to 28 by 28 pixels. Once again, you see that the way you actually form the list: for every image that you find in the images array, you'll perform the transformation operation that you borrow from the skimage library. Finally, you store the result in the images28 variable:

# Import the `transform` module from `skimage`
from skimage import transform

# Rescale the images in the `images` array
images28 = [transform.resize(image, (28, 28)) for image in images]

This was fairly easy wasn't it?

Note that the images are now four-dimensional: if you convert images28 to an array and if you concatenate the attribute shape to it, you'll see that the printout tells you that images28's dimensions are (4575, 28, 28, 3). The images are 784-dimensional (because your images are 28 by 28 pixels).

You can check the result of the rescaling operation by re-using the code that you used above to plot the 4 random images with the help of the traffic_signs variable; Just don't forget to change all references to images to images28.

Check out the result here:

Note that because you rescaled, your min and max values have also changed; They seem to be all in the same ranges now, which is really great because then you don't necessarily need to normalize your data!

As said in the introduction to this section of the tutorial, the color in the pictures matters less when you're trying to answer a classification question. That's why you'll also go through the trouble of converting the images to grayscale.

Note, however, that you can also test out on your own what would happen to the final results of your model if you don't follow through with this specific step.

Just like with the rescaling, you can again count on the Scikit-Image library to help you out; In this case, it's the color module with its rgb2gray() function that you need to use to get where you need to be.

That's going to be nice and easy!

However, don't forget to convert the images28 variable back to an array, as the rgb2gray() function does expect an array as an argument.

# Import `rgb2gray` from `skimage.color`
from skimage.color import rgb2gray

# Convert `images28` to an array
images28 = np.array(images28)

# Convert `images28` to grayscale
images28 = rgb2gray(images28)

Double check the result of your grayscale conversion by plotting some of the images; Here, you can again re-use and slightly adapt some of the code to show the adjusted images.

Note that you indeed have to specify the color map or cmap and set it to "gray" to plot the images in grayscale. That is because imshow() by default uses, by default, a heatmap-like color map. Read more here.

Tip: since you have been re-using this function quite a bit in this tutorial, you might look into how you can make it into a function :)

These two steps are very basic ones; Other operations that you could have tried out on your data include data augmentation (rotating, blurring, shifting, changing brightness,...). If you want, you could also set up an entire pipeline of data manipulation operations through which you send your images.

Deep Learning With TensorFlow

Now that you have explored and manipulated your data, it's time to construct your neural network architecture with the help of the TensorFlow package!

# data[].attributes.attachment_link.title
TensorFlow Tutorial For Beginners - Hacker Noon

# data[].attributes.attachment_link.description
Originally published at https://www.datacamp.com/community/tutorials/tensorflow-tutorial Deep learning is a subfield of machine learning that is a set of algorithms that is inspired by the structure and function of the brain.

# data[].attributes.attachment_link.readable_text
Xing Wang Mar 16

note, this doc is written by my co-worker Derric, but we blog at the same place.

Since being introduced by Facebook, GraphQL has taken the API world by storm as an alternative to REST APIs. GraphQL fixes many problems that API developers and users have found with RESTful architecture. However, it also introduces a new set of challenges which need to be evaluated. Because GraphQL is not simply a evolutionary replacement for RESTful APIs, this post will deep dive into the pros and cons of each and when GraphQL makes sense for your application.

History

Before RESTful APIs, we had RPC, SOAP, CORBA, and other less open protocols. Many pre-REST APIs required complex client libraries to serialize/deserialize the payload over the wire. A fundamental difference compared to today's RESTful APIs, is that SOAP is strongly typed using formal contracts via WSDL (Web Services Description Language). This could wreck havoc on interoperability since even removing a 32-bit integer restriction to accepting a 64-bit long meant breaking upstream clients. SOAP is not an architecture, but a full protocol implementation consisting of security, error handling, ACID transactions, etc Some of the complexity is due to many abstraction layers baked into SOAP. For example, SOAP is able to run on HTTP, TCP, UDP, etc. While the protocol implementation afforded much abstraction, the application and data layer's rigid contract between the client and server created a tight coupling between the two.

RESTful architecture was introduced in 2000 as a much simpler way to enable machine to machine communication using only the ubiquitous HTTP protocol without additional layers in a stateless and type free way. This enabled systems to be loosely coupled and more forgiving to contract changes between systems such as between companies.

REST today

REST APIs have become the de facto standard for companies deploying APIs and launching developer platforms. The beauty of REST is that a developer working with someone else's API doesn't need any special initialization or libraries. Requests can be simply sent via common software like cURL and web browsers.

REST uses the standard CRUD HTTP Verbs (GET, POST, PUT, DELETE) and leverages HTTP conventions and centered around data resources (HTTP URIs) rather than attempting to fight HTTP. Thus an e-commerce with a resource api.acmestore.com/items can behave similar to a bank with a resource api.examplebank.com/deposits. Both would probably need CRUD operations on that resource and prefer to cache queries (i.e. both GET api.acmestore.com/items and GET api.examplebank.com/transactions would be cached). 3rd party developers to a new API need to only reason about the data model and leave the rest to HTTP convention rather than digging deep into thousands of operations. In other words, REST is much tighter coupled to HTTP and CRUD compared to SOAP, but provides loose data contracts.

Problems with REST

As more variety of APIs are placed in production use and scaled to extreme levels, certain problems in RESTful architecture transpired. You could even say GraphQL is between SOAP and REST taking pieces from each.

Server driven selection

In RESTful APIs, the server creates the representation of a resource to be responded back to a client.

However, what if the client wants something specific such as return the names of friends of friends of a user where their job is engineer.

With REST, you might have something like:

GET api.example.com/users/123?include=friend.friend.name&friend.friend.ocupation=engineer

GraphQL allows you to represent this query in a cleaner way:

{ user(id: 123) { friends { friends(job: "engineer") { name } } } }

Fetching multiple resources

One of the main benefits of GraphQL is to make APIs less chatty. Many of us have seen an API where we first have to GET /user first and then fetch each friend individually via GET /user/:id/friend/:id endpoint, this can result in N+1 queries and is a well known performance issue in API and database queries. In other words, RESTful API calls are chained on the client before the final representation can be formed for display. GraphQL can reduce this by enabling the server to aggregate the data for the client in a single query.

More in depth analytics

While API analytics is also a negative for GraphQL apis since there is very little tooling out there. The tools that do support GraphQL APIs can provide much more insights into queries than RESTful APIs.

Problems with GraphQL

Caching

Caching is built into in the HTTP specification which RESTful APIs are able to leverage. GET vs POST semantics related to caching are well defined enabling browser caches, intermediate proxies, and server frameworks to follow. The following guidelines can be followed:

GET requests can be cached

GET requests can stay in browser history

GET requests can be bookmarked

GET requests are idempotent

GraphQL doesn't follow the HTTP spec for caching and instead uses a single endpoint. Thus, it's up to the developer to ensure caching is implemented correctly for non-mutable queries that can be cached. The correct key has to be used for the cache which may include inspecting the body contents.

While you can use tools like Relay or Dataloader that understands GraphQL semantics, that still doesn't cover things like browser and mobile caching.

Diminishes shared nothing architecture

The beauty of RESTful APIs is that they complement shared nothing architecture well. For example, Moesif has a api.moesif.com/v1/search endpoint and a api.moesif.com/v1/alerting endpoint. Publicly, those two endpoints simply look like two different REST resources. Internally though, they point to two different microservices on isolated compute clusters. The search service is written in Scala and the alerting service is written in NodeJS. The complexity in routing HTTP requests via host or URL is much lower than inspecting a GraphQL query and performing multiple joins.

Exposed for arbitrary requests

While a main benefit of GraphQL is to enable clients to query for just the data they need, this can also be problematic especially for open APIs where an organization cannot control 3rd party client query behavior. Great care has to be taken to ensure GraphQL queries don't result in expensive join queries that can bring down server performance or even DDoS the server. RESTful APIs can be constrained to match data model and indexing used.

Rigidness of queries

GraphQL removes the ability for custom query DSLs or side effect operations on top of an API. For example, the Elasticsearch API is RESTful, but also has a very powerful Elasticsearch DSL to perform advanced aggregations and metric calculations. Such aggregation queries may be harder to model within the GraphQL language.

Non existent monitoring

RESTful APIs have the benefit of following the HTTP spec with regards to resources just like a website. This enables many tools to probe a URL such as api.moesif.com/health which would return 5xx if not OK. For GraphQL APIs, you may not be able to leverage such tools unless you support placing the query as a URL parameter as most ping tools don't support HTTP and request bodies.

Besides ping services, there are very few SaaS or open source tools that support API analytics or deeper analysis of your API calls. Client errors are presented as a 200 OK in a GraphQL API. Existing tools that expect 400 errors will not work so you may miss errors happening on your API. Yet at the same time, more flexibility given to the client requires even more tools to catch and understand problems with your API.

Conclusion

GraphQL APIs can be exciting new technology, but it is important to understand the tradeoffs before making such architectural decisions. Some APIs such as those with very few entities and relationships across entities like analytics APIs may not be suited for GraphQL. Whereas applications with many different domain objects like e-commerce where you have items, users, orders, payments, and so on may be able to leverage GraphQL much more.

In fact, GraphQL vs REST is like comparing SQL technologies vs noSQL. There are certain applications where it makes sense to model complex entities in a SQL Db. Whereas other apps that only have "messages" as in high volume chat apps or analytics APIs where the only entity is an "event" may be more suited using something like Cassandra.

But at the same time, we expect the usage of GraphQL to continue rise, and the problems can be addressed with better tooling. For GraphQL, in-depth analytics make even more sense, check out our support for GraphQL at Moesif.

Did you find this post useful? Show some love!

# data[].attributes.attachment_link.title
REST vs GraphQL APIs, the Good, the Bad, the Ugly

# data[].attributes.attachment_link.description
note, this doc is written by my co-worker Derric, but we blog at the same place. Since being introduced by Facebook, GraphQL has taken the API world by storm as an alternative to REST APIs. GraphQL fixes many problems that API developers and users have found with RESTful architecture.

# data[].attributes.attachment_link.readable_text
I was recently chatting to a friend whose startup's machine learning models were so disorganized it was causing serious problems as his team tried to build on each other's work and share it with clients. Even the original author sometimes couldn't train the same model and get similar results! He was hoping that I had a solution I could recommend, but I had to admit that I struggle with the same problems in my own work. It's hard to explain to people who haven't worked with machine learning, but we're still back in the dark ages when it comes to tracking changes and rebuilding models from scratch. It's so bad it sometimes feels like stepping back in time to when we coded without source control.

When I started out programming professionally in the mid-90's, the standard for keeping track and collaborating on source code was Microsoft's Visual SourceSafe. To give you a flavor of the experience, it didn't have atomic check-ins, so multiple people couldn't work on the same file, the network copy required nightly scans to avoid mysterious corruption, and even that was no guarantee the database would be intact in the morning. I felt lucky though, one of the places I interviewed at just had a wall of post-it notes, one for each file in the tree, and coders would take them down when they were modifying files, and return them when they were done!

This is all to say, I'm no shrinking violet when it comes to version control. I've toughed my way through some terrible systems, and I can still monkey together a solution using rsync and chicken wire if I have to. Even with all that behind me, I can say with my hand on my heart, that machine learning is by far the worst environment I've ever found for collaborating and keeping track of changes.

To explain why, here's a typical life cycle of a machine learning model:

A researcher decides to try a new image classification architecture.

She copies and pastes some code from a previous project to handle the input of the dataset she's using.

This dataset lives in one of her folders on the network. It's probably one of the ImageNet downloads, but it isn't clear which one. At some point, someone may have removed some of the images that aren't actually JPEGs, or made other minor modifications, but there's no history of that.

She tries out a lot of slightly different ideas, fixing bugs and tweaking the algorithms. These changes are happening on her local machine, and she may just do a mass file copy of the source code to her GPU cluster when she wants to kick off a full training run.

She executes a lot of different training runs, often changing the code on her local machine while jobs are in progress, since they take days or weeks to complete.

There might be a bug towards the end of the run on a large cluster that means she modifies the code in one file and copies that to all the machines, before resuming the job.

She may take the partially-trained weights from one run, and use them as the starting point for a new run with different code.

She keeps around the model weights and evaluation scores for all her runs, and picks which weights to release as the final model once she's out of time to run more experiments. These weights can be from any of the runs, and may have been produced by very different code than what she currently has on her development machine.

She probably checks in her final code to source control, but in a personal folder.

She publishes her results, with code and the trained weights.

This is an optimistic scenario with a conscientious researcher, but you can already see how hard it would be for somebody else to come in and reproduce all of these steps and come out with the same result. Every one of these bullet points is an opportunity to inconsistencies to creep in. To make things even more confusing, ML frameworks trade off exact numeric determinism for performance, so if by a miracle somebody did manage to copy the steps exactly, there would still be tiny differences in the end results!

In many real-world cases, the researcher won't have made notes or remember exactly what she did, so even she won't be able to reproduce the model. Even if she can, the frameworks the model code depend on can change over time, sometimes radically, so she'd need to also snapshot the whole system she was using to ensure that things work. I've found ML researchers to be incredibly generous with their time when I've contacted them for help reproducing model results, but it's often months-long task even with assistance from the original author.

Why does this all matter? I've had several friends contact me about their struggles reproducing published models as baselines for their own papers. If they can't get the same accuracy that the original authors did, how can they tell if their new approach is an improvement? It's also clearly concerning to rely on models in production systems if you don't have a way of rebuilding them to cope with changed requirements or platforms. At that point your model moves from being a high-interest credit card of technical debt to something more like what a loan-shark offers. It's also stifling for research experimentation; since making changes to code or training data can be hard to roll back it's a lot more risky to try different variations, just like coding without source control raises the cost of experimenting with changes.

It's not all doom and gloom, there are some notable efforts around reproducibility happening in the community. One of my favorites is the TensorFlow Benchmarks project Toby Boyd's leading. He's made it his team's mission not only to lay out exactly how to train some of the leading models from scratch with high training speed on a lot of different platforms, but also ensures that the models train to the expected accuracy. I've seen him sweat blood trying to get models up to that precision, since variations in any of the steps I listed above can affect the results and there's no easy way to debug what the underlying cause is, even with help from the authors. It's also a never-ending job, since changes in TensorFlow, in GPU drivers, or even datasets, can all hurt accuracy in subtle ways. By doing this work, Toby's team helps us spot and fix bugs caused by changes in TensorFlow in the models they cover, and chase down issues caused by external dependencies, but it's hard to scale beyond a comparatively small set of platforms and models.

I also know of other teams who are serious about using models in production who put similar amounts of time and effort into ensuring their training can be reproduced, but the problem is that it's still a very manual process. There's no equivalent to source control or even agreed best-practices about how to archive a training process so that it can be successfully re-run in the future. I don't have a solution in mind either, but to start the discussion here are some principles I think any approach would need to follow to be successful:

Researchers must be able to easily hack around with new ideas, without paying a large "process tax". If this isn't true, they simply won't use it. Ideally, the system will actually boost their productivity.

If a researcher gets hit by a bus founds their own startup, somebody else should be able to step in the next day and train all the models they have created so far, and get the same results.

There should be some way of packaging up just what you need to train one particular model, in a way that can be shared publicly without revealing any history the author doesn't wish to.

To reproduce results, code, training data, and the overall platform need to be recorded accurately.

I've been seeing some interesting stirrings in the open source and startup world around solutions to these challenges, and personally I can't wait to spend less of my time dealing with all the related issues, but I'm not expecting to see a complete fix in the short term. Whatever we come up with will require a change in the way we all work with models, in the same way that source control meant a big change in all of our personal coding processes. It will be as much about getting consensus on the best practices and educating the community as it will be about the tools we come up with. I can't wait to see what emerges!

# data[].attributes.attachment_link.title
The Machine Learning Reproducibility Crisis

# data[].attributes.attachment_link.description
Gosper Glider Gun I was recently chatting to a friend whose startup's machine learning models were so disorganized it was causing serious problems as his team tried to build on each other's work and share it with clients. Even the original author sometimes couldn't train the same model and get similar results!

# data[].attributes.attachment_link.readable_text
You don't need a virtual machine to run your own source code any more as AWS Lambda offers execution environments for Java, JavaScript (Node.js), C#, Go, and Python. All you need to do is to implement a function, upload your source code and configure the execution environment. Afterwards, your source code is executed within a fully-managed computing environment. Your new tool enables you to automate operations tasks within your infrastructure with ease. This is because AWS Lambda is well-integrated with all parts of AWS. We use AWS to automate our infrastructure regularly. For example, to add and remove instances to a container cluster, based on a custom algorithm, or to process and analyze log files.

AWS Lambda offers a maintenance-free and highly-available computing environment. You no longer need to install security updates, replace failed virtual machines, or manage remote access (SSH or RDP) for administrators. On top of that, AWS Lambda isn't billed by the second but per execution. Therefore, you don't need to pay for idling resources which are waiting for work (e.g. a task triggered once a day).

AWS Lambda is a powerful tool in different scenarios: a REST API for a web or mobile application, an event-driven system to process data, or even an IoT backend. We are also big fans of using AWS Lambda to automate operational tasks within our cloud infrastructure. For example, using a Lambda function to perform periodic health checks of a website. Or to add tags to newly launched EC2 instances automatically.

But what is AWS Lambda? We'll start with a short introduction.

This article, excerpted from chapter 7 of Amazon Web Services in Action, Second Edition, is about adding a new tool to your toolbox. The tool we're talking about is as flexible as a Swiss army knife, and it's called AWS Lambda.

Save 37% offAmazon Web Services in Action, Second Edition with codefccwittig atmanning.com.

Executing Your Code with AWS Lambda

Compute capacity is available at different layers of abstraction on AWS: virtual machines, containers, and functions. Containers offer another layer of abstraction on top of virtual machines. Unfortunately, we don't cover containers in this article. AWS Lambda provides computing power as well but in a fine-granular manner: an execution environment for small functions, not a full-blown operating system or container.

What Is Serverless?

When reading about AWS Lambda you might have stumbled upon the term serverless already. The following quote summarizes the confusion created by a catchy and provocative phrase:

[... ] the word serverless is a bit of a misnomer. Whether you use a computer service such as AWS Lambda to execute your code, or interact with an API, there are still servers running in the background. The difference is that these servers are hidden from you. No infrastructure exists for you to think about and there's no way to tweak the underlying operating system. Someone else takes care of the nitty-gritty details of infrastructure management, freeing your time for other things.
Peter Sbarski, Serverless Architectures on AWS, Manning (2017)

We define the following the following criteria for a serverless system:

No need to manage and maintain virtual machines.

Fully managed service offering scalability and high availability.

Billed per request and its resource consumption.

AWS isn't the only provider offering a serverless platform. Google (Cloud Functions) and Microsoft (Azure Functions) are competing with AWS in this area, for example.

Running Your Code on AWS Lambda

As shown in the following figure, to execute your code with AWS Lambda the following four steps are needed:

1. Writing the source code.

2. Uploading your code and its dependencies (e.g. libraries or modules).

3. Creating a function determining the runtime environment and configuration.

4. Executing the function.

You don't need to launch any virtual machines. AWS executes your code in a fully-managed compute environment.

Currently, AWS Lambda offers runtime environments for the following languages:

Next, you'll learn more about AWS Lambda by comparing its concepts with virtual machines offered by EC2.

Comparing AWS Lambda with Virtual Machines (Amazon EC2)

What's the difference between AWS Lambda and virtual machines? First off, the granularity of virtualization. On the one hand, a virtual machine provides a full operating system which is used to run one or multiple applications. On the other hand, AWS Lambda offers an execution environment for a function, a small part of an application.

Furthermore, Amazon EC2 offers virtual machines as a service. But you're still responsible for operating these virtual machines in a secure, scalable and highly available way. Doing this generates a bunch of operating tasks resulting in substantial maintenance effort. By contrast, AWS Lambda offers a fully-managed execution environment. AWS is managing the underlying infrastructure for you and provides a production-ready infrastructure to you.

Otherwise AWS Lambda is billed per execution, not per second that a virtual machine is running. You don't need to pay for unused resource waiting for requests or tasks any longer. For example, running a script to check the health of a website every five minutes on a virtual machine costs you a minimum of USD 4. Executing the same health check with AWS Lambda is free as you don't even exceed the never-ending monthly free tier.

The following table compares AWS Lambda and virtual machines in detail.

Learn More

Do you want to learn more? Check out chapter 7 of our book Amazon Web Services in Action, Second Edition including two real-world examples: website health check and automatically tagging a newly launched EC2 instance with it's owner.

Also, this slide deck introduces the rest our our book in more detail.

# data[].attributes.attachment_link.description
You don't need a virtual machine to run your own source code any more as AWS Lambda offers execution environments for Java, JavaScript (Node.js), C#, Go, and Python. All you need to do is to implement a function, upload your source code and configure the execution environment.

# data[].attributes.attachment_link.readable_text
This is part of "Blockchain Decoded," a series looking at the impact of blockchain, bitcoin and cryptocurrency on our lives.

These days, we're having a harder and harder time trusting each other.

Trust is an essential part of ordinary living, whether it's picking mechanics based on Yelp reviews, sliding credit cards into gas station fuel pumps or heeding our doctor's advice. But our trust has been eroding for years. In the US, only 33 percent of us felt we could trust our government in 2017 -- a decline of 14 percentage points from 2016, according to Edelman's annual trust barometer study. Trust in businesses dropped from 58 percent to 48 percent, too, while media (fake news!) and social networks also took a hit.

That's a problem. The less trust you have, the harder everything becomes. Did that job candidate really graduate from college? Did your brother-in-law really repay that loan?

But there's an unlikely solution that might help restore enough faith in strangers to make our lives a bit easier: an encryption technology called blockchain.

Blockchain is best known as the technology behind the cryptocurrency -- a digital currency whose value soared above $19,000 over the last year before slumping to half that when the frenzy subsided. But blockchain is so much more, potentially easing the doubts and uncertainties that dog so much of life -- whether buying a used car from a stranger, having faith that a piece of fruit really is organic or knowing that a prescription drug isn't counterfeit. Blockchain, in effect, hard-wires trust into transactions or data that we might otherwise be more cautious about.

"It's revolutionary," said Mark Siegel, an investor at Menlo Ventures.

Bitcoin's value has soared and plunged over the last year, and it's hard to separate the sensible from the among the 1,500 other cryptocurrencies. But blockchain has enjoyed more stable appeal.

Indeed, staid companies like IBM, Microsoft and Intel are offering blockchain as just another software tool to get business done. Other companies dabbling in blockchain include Goldman Sachs, Nasdaq, Walmart and Visa.

Because blockchains work as a secure digital ledger, a bumper crop of startups are hoping to bring it to voting, lotteries, ID cards and identity verification, graphics rendering, welfare payments, job hunting and insurance payments.

A lot of that revolution could be invisible to you, taking place inside and among businesses. But it's potentially a very big deal. Analyst firm Gartner estimates that blockchain will provide $176 billion in value to businesses by 2025 and a whopping $3.1 trillion by 2030.

How does blockchain actually work?

OK, strap yourself in, because this gets a bit hairy.

A good place to start is the name: a blockchain is an ever-growing set of data blocks. Each block records a collection of transactions -- for example, that you now hold the title to the car you bought or that you paid a car dealer to get it.

That may sound simple, but here's a difference between blockchain and the Department of Motor Vehicles. Today, the government stores the information on its own central computer. Blockchains, though, distribute it across a group of computers -- maybe even thousands of them. Each has its own copy of the blockchain transactions.

That decentralization and synchronization means no single party controls the data. If one business sells an asset to another, each sees the same data. There's no need for lawyers at one company to call the other if their accounting databases disagree, because there's only one accounting database.

Cryptography -- mathematical methods of keeping data secret and proving identity -- now enters the picture when it comes to recording transactions. Blockchain uses the same cryptographic key technology that keeps hackers from sniffing your credit card number when you type it into an e-commerce website. One digital key ensures only you can enter a transaction to the blockchain involving your assets, and another digital key lets someone else confirm it really was you who added the transaction.

"You can take a network of parties that didn't have prior experience working with each other -- that didn't have reason for trust -- and still find a way to build a transaction record or a history of the truth," said Brian Behlendorf, executive director for the Linux Foundation's Hyperledger project for blockchain software.

Indelible ink

Another fundamental part of the blockchain is called immutability -- its resistance to tampering or other changes. To understand it, you need to understand another cryptographic concept called the hash.

Hashing reduces data to a bunch of seemingly random characters -- for example, the hash of the phrase "the quick brown fox" is "9ECB36561341D18EB65484E833EFEA61EDC74B84CF5E6AE1B81C63533E25FC8F" using an encoding method called SHA-256. Tweaking just one letter in the phrase produces a completely different hash, and you can't go backward to figure out the original data from the hash.

With blockchain, hashes are linked together so any minute change is immediately visible, not just for the block housing it but for all other blocks added later. With red flags that big for changes that small, you can see why auditors would get excited.

"It's like doing the crossword puzzle in ink instead of pencil," said Marie Wieck, head of IBM's 1,500-employee blockchain group. "You will see if you change your answer to 3 across from moon to star."

That's no fun for embezzlers accustomed to hiding behind dodgy or altered records. Cryptocurrencies can offer anonymity to criminals, which is why it's been popular for things like the WannaCry ransomware that locked up people's computers until they paid up. But blockchain makes it easier to find the digital scene of the crime -- especially with private blockchains that networks of business partners can set up to cooperate.

Mining madness

The process for locking down a block onto the blockchain so it can't be changed, at least today, is called mining.

And it's a problem.

Here's how it works. When you and others announce transactions to a blockchain network, computers on that network race to solve a complicated mathematical puzzle based on those transactions. A computer that succeeds announces it to the network, and the transaction is accepted if other computers verify that none of the assets in question were already used. That's what'll keep you from selling the same concert ticket twice on a blockchain-based ticket market. ( Citizen Ticket and Active Ticketing are working on this.)

But today's mining approach, called "proof of work," has huge drawbacks.

For one thing, mining works most profitably on powerful computers that consume immense amounts of electrical power. For example, bitcoin mining today uses about as much power as the country of Singapore, enough to power 4.4 million houses, according to cryptocurrency analyst firm Digiconomist. That amount is growing.

For another, transactions are relatively slow. Blockchain transactions can race past transactions that rely on middlemen and reconciliation procedures, like escrow accounts for home purchases or international money transfers. But bitcoin transactions can take about 10 minutes, which is why cryptocurrencies today aren't useful for just buying something in a store.

There's lots of work to free blockchain from the problems of transaction speed and energy consumption, though. One idea, "proof of stake," uses no significant computing power and looks to be the future for the Ethereum Project, which is responsible for the ether cryptocurrency.

If bitcoin was the first generation of blockchain and the second, there are a number of people hoping their project will catch on as the third.

Tezos, for example, hopes to build in better governance so its technology can move forward without the troubles bitcoin and Ethereum have suffered, said Tezos CEO Kathleen Breitman, speaking at the Techonomy conference in November -- though ironically, Tezos has suffered governance problems of its own with a spat over its own management. Another challenger is Dfinity. Its chief scientist, Dominic Williams, promises transaction speeds 600 times faster than Ethereum, which today is only a bit faster than bitcoin.

Smart contracts

The original blockchain was described in a 2008 bitcoin paper by Satashi Nakamoto, a pseudonym for a person or perhaps group that unified some ideas into the first working cryptocurrency. The idea became reality with the release of open-source bitcoin software in 2009. The bitcoin blockchain now records about 300 million transactions and counting.

But ether has popularized a newer idea called smart contracts. These are programs that run on the Ethereum network and take automated if-this-then-that actions. For example, a smart contract could look for the highest bid in an auction at a certain time and automatically transfer ownership rights to the auction winner.

"When companies sign a contract, it's enforced by a judge or lawyers in a court," said Vipul Goyal, an associate professor in Carnegie Mellon University's cryptography group. "Smart contracts are enforced by cryptographic mechanisms in the code. Enforcing the contract is much cheaper and much faster -- almost instant."

With smart contracts, blockchain could help automate lots of computing operations, including ones humans never touch. Your electric car could wait for favorable electricity prices before deciding when to charge itself from the grid, solar panels or in-home batteries, then the blockchain could handle accounting among all the parties.

Goyal expects blockchain will help automate all sorts of transactions. For example, if it's used to register your car purchase, that could trigger a cascade of other operations, like transferring the car's cryptographic keys that let its owner unlock the car.

"This is much more efficient than going to the DMV and filling out paperwork," he said. "It's also more secure, because these keys cannot be forged. The seller can't make copies of the key and try to steal the car."

The ties that bind

Expect to see blockchain showing up in particular where there are groups of interlinked organizations. That could include one company and its suppliers, or it could be consortiums of competitors and and their suppliers.

For example, IBM has a blockchain partnership with a long list of food suppliers and grocery retailers, including Dole, Kroger, Nestlé, Tyson Foods and Walmart.

Another blockchain project comes through browser startup Brave, which relies on the technology to change online advertising in a way that improves performance and privacy while giving browser users a cut of the proceeds. Blockchain accounting, using a digital payment mechanism called the basic attention token (BAT), enables direct payments among advertisers, publishers and browser users -- for example an advertiser paying a publisher or a reader making a small one-off payment for a news article without buying a subscription.

It's transparent, so anyone can see exactly how many BATs were transferred and check that Brave didn't illicitly siphon any off, Brave CEO Brendan Eich said.

But for companies averse to sharing data with competitors, blockchain's transparency is a difficulty. There are mechanisms for handling the challenge, Behlendorf said.

"In most networks, you have a balance between data that can be kept private, but enough public that you can attest to its veracity," Behlendorf said.

Another way blockchain could bring many parties together is property records.

There are thousands of counties in the US, each with its own record of who owns what. One startup, Propy, hopes to digitize those records, mirroring the records initially the way title companies do, but also storing them on the blockchain, said CEO Natalia Karayaneva.

If county clerks saw the benefit, they could gradually move to the system -- it's decentralized, not Propy's own database. Propy hopes to profit by taking a percentage of the sales it facilitates, but at the same time, it also hopes to cut purchasers' costs -- for example by eliminating the thousands of dollars that title insurance can cost.

Slow down there a minute

For something as hyped as blockchain, with millions of dollars raised, you have to expect some backlash. There's plenty, starting with the criticism that blockchain would have already taken off if it's so great and concerns that it's abetting cryptocurrency shenanigans. There's also the concern that poorly written code could leave a faulty foundation.

Overinflated expectations are nothing new to the tech industry, though, and there are enough serious players engaged that it's hard to dismiss blockchain as all sizzle and no steak. Expect a winnowing as reality sets in.

"In 2018, we expect to see a number of projects stopped that should never have been started in the first place," said Forrester analyst Martha Bennett.

She points out plenty of other areas where blockchain falls short of its promises. The immutability comes at a cost, lacking some of the mechanisms for recourse found in today's slower processes. Companies cooperating to set up their own private blockchains, rather than using public ones like Ethereum, must have some trust already to set up rules for access and governance.

Here's another hitch: getting everybody on board. The French automaker Renault, for example, hopes for a blockchain to lock down car maintenance records. After all, who wouldn't want to know if the used car you're thinking of buying made lots of trips to the repair shop? It turns out the seller may not share your enthusiasm for that much transparency.

So it's not perfect. But it doesn't have to be. Blockchain just has to be better than what we have today. There are a lot of underhanded cryptocurrency dealings, but regulators are now reining in abuses, said Rick Levin, chairman of the financial technology and regulation team at the AmLaw law firm Polsinelli. Likewise, engineers are hammering out improvements to blockchain and big names like Nasdaq and Goldman Sachs are embracing it.

"I don't think it's just going to vanish," Levin said. "There's too much energy behind this."

The Smartest Stuff: Innovators are thinking up new ways to make you, and the things around you, smarter.

CNET Magazine: Check out a sample of the stories in CNET's newsstand edition.

# data[].attributes.attachment_link.title
Why should you care about blockchain? It's the ultimate trust builder

# data[].attributes.attachment_link.description
This is part of "Blockchain Decoded," a series looking at the impact of blockchain, bitcoin and cryptocurrency on our lives. These days, we're having a harder and harder time trusting each other. Trust is an essential part of ordinary living, whether it's picking mechanics based on Yelp reviews, sliding credit cards into gas station fuel pumps or heeding our doctor's advice.

# data[].attributes.attachment_link.readable_text
With all the talk about artificial intelligence (AI) and machine learning (ML) doing crazy things, it's easy to be left wondering, "what are practical ways I can use this today?" It turns out there are some extremely easy ways to try this today.

In this post, I'll walk through how to detect faces, gender, ages, and hair color in photos, by adding only a few lines of code to an ASP.NET app. Images will be uploaded and shown in an image gallery built with ASP.NET, images will be hosted in Azure Storage, and Azure Cognitive Services will be used to analyze the images. The full application is available on GitHub. To begin, clone the repository on your machine.

What we'll build

Here's what the recognized photos can look like when displayed in a web browser. Note how the image and metadata generated by Azure Cognitive Services is displayed alongside it.

Set up prerequisites with Visual Studio and Azure

To begin, make sure you've installed Visual Studio 2017 with the ASP.NET and web workload. This will provide everything you need to build and run the app yourself.

Next, set up the Azure prerequisites.

First, ensure you have an Azure account. If not, you can sign up for an Azure free account, which will give you a $200 credit towards anything.

Next, create a Storage account, through the Azure Portal:

You'll need to create the Storage resource:

After creating the resource, you'll need to create the storage account for your resource with the default settings:

Finally, create a Cognitive Services resource through the Azure portal:

Once you've set that up, you're ready to start hacking away at the sample app!

Explore the codebase

Open the project in Visual Studio 2017 if you haven't already. The application is an ASP.NET MVC app. It does three major things:

The first major operation is uploading an image to Azure Blob storage, analyzing the image using Azure Cognitive Services, and uploading image metadata generated from Cognitive Services back to Blob Storage.

The second major operation is to snag images and their associated metadata from Blob Storage.

The UI simply wires up these images to a page with an upload button.

Add your API keys

Modify the Web.config file to include your Cognitive Services URL and Cognitive Services API key. Look for this file:

Your Cognitive Services URL and API keys can be found in the dashboard for your Cognitive Services resource in the Azure Portal here:

The connection strings for your Azure Storage resource can be found in the Azure Portal under Access Keys:

Once you have entered your information in your Web.Config file, you'll be good to go!

To learn more about how to best work with keys and other information in a development, see Best Practices for Deploying Passwords and other Sensitive Data to ASP.NET and Azure.

Run the application and add some images

Now that everything is set up and configuration has been set up locally, you can run the application on your machine!

Press F5 to debug and see how everything works. I recommend that you set a breakpoint in the Upload controller action ( HomeController.cs, line 32), so that you can step through each operation as you upload a new image. In the opened browser, upload an image to see what happens!

If you want to see images show up in Azure blobs when running the app, you can do so with Cloud Explorer ( View -> Cloud Explorer). You may need to log in first, but after that, you can navigate to your created Storage Account and see all of your Blobs under Blob Containers:

In this example, I've uploaded three images to my container called "images". The web app also uploaded a json file with image metadata for each image.

Publish to Azure and impress your friends with your use of AI

You can publish the entire application to Azure App Service. Right-click on your project and select "Publish". Next, select App Service and continue. You can create one right in the Visual Studio UI:

Finally, click "Create" and it will create all the Azure resources you need and publish your app! After that process completes (it should take a minute or two), your browser will open with your application running entirely in Azure.

Next steps

And that's it! Try exploring other, interesting things you can do with Cognitive services. Some fun things to try, without needing to add support for any services or read other tutorials:

Modify the web app to replace someone's face with an emoji that matches their measured emotion (try the System.Drawing API!)

Group faces by similarity, age, or if they have makeup on

Try it out on pictures of animals instead of humans

Additionally, check out these tutorials to learn more about what you can do with .NET and Cognitive Services:

Cheers, and happy coding!

# data[].attributes.attachment_link.title
Learn how to do Image Recognition with Cognitive Services and ASP.NET

# data[].attributes.attachment_link.description
With all the talk about artificial intelligence (AI) and machine learning (ML) doing crazy things, it's easy to be left wondering, "what are practical ways I can use this today?" It turns out there are some extremely easy ways to try this today.

# data[].attributes.attachment_link.readable_text
PHOTO: Shutterstock

The software products we use every day for both business and leisure are going through a quiet transition behind the scenes. A decade ago, it was the norm to build software as one, large, unified product - as a monolith. But in recent years, there has been more adoption of microservices, or microservice architecture, which constitutes a drastically different approach to building software.

What Are Microservices?

With a microservices, a type of service oriented architecture, software is built in smaller loosely coupled segments, rather than one large monolith. By segmenting key parts of a software, you can silo and decentralize a suite of software products that work independently, but they can also work in tandem when necessary. It also allows for smaller more focused teams that can work more independently. "Microservice architecture is an approach to developing a single application as a suite of small services, rather than as one large product," he says. These individual services run in their own process and communicate with each other via lightweight mechanisms, often in the form of an HTTP resource API," says Jake Lumetta, CEO of Chicago based ButterCMS.

Microservice Pros and Cons

If you're considering microservice architecture for your next project, here's a summary of the pros and cons of microservices.

Microservice Pros:

Better Organization: Microservice architecture usually results in a more organized development environment. Everything has its place - but they keyword here is usually, as microservices can get just as messy as a monolith.

Better Performance: Under the right circumstances, microservices can also have performance advantages depending on how they're organized because it's possible to isolate hot services and scale them independent of the rest of the app.

Leaner, More Targeted Teams: With microservice architecture, a small team is often in charge of one or a small group of services, which can enhance agile and SCRUM development strategies.

Easier to Scale: Microservices are highly scalable, and easy to use with containerization technology such as Docker. Plus, you can scale parts of your software product independently.

Decoupled Services: Microservices are decoupled, which means they can be modified independently and can individually serve the purposes of different apps.

Microservices Cons:

Higher Overheads: Microservices typically take up their own virtual machines or containers, which can lead to higher overhead costs.

Frequent Communication Needed: When apps get siloed, teams can sometimes follow suit. Thus, developers need to compensate by communicating frequently so that everybody understands the bigger picture.

The Difference Between Microservices and Monoliths

San Francisco, Calif.-based Dialpad's Co-Founder and VP of Telephony, John Rector offers this befitting metaphor. "Microservices is an architecture design for building applications, particularly of the cloud/SaaS variety. I think the best way to understand the approach is by using an example, contrasting with the old monolithic approach," Rector says.

He went on to describe how a startup phone service provider may launch using monolithic architecture.

"[Say an entrepreneur] in the phone service industry builds a prototype, where everything runs on his laptop: the database, the webserver, the telephony server - everything. VCs like the demo, so he takes the prototype and puts it on a cloud server. Eventually he signs a lot of customers, only to find out his server can handle just up to 1,000 users. So, he sets up nine more of those same servers, throws a load-balancer in front, and now he can handle 10,000 users. By this time he's received some reliability complaints, so he sets up his servers with a hot standby for each one: if machine A crashes, there's a duplicate A' which is ready to start handling traffic. This is the old monolithic architecture: everything runs on one machine, and if you need to scale up, you duplicate the whole system with more machines," he says.

The alternative, is a microservices architecture. In this new example, Rector explains how the entrepreneur resists the urge to lump everything on one laptop or server.

"The big difference [with the microservices approach] is that the database, the webserver and the telephony server talk to each other using APIs (basically, HTTP requests). At first they all run on the same machine, but when he adds more hardware, he pulls each type of server onto its own machine: one Dell box for the webserver, another for the database, and they talk to each other with HTTP," Rector told CMSWire.

Eventually, Rector explains, the entrepreneur will gain too much traffic for that one webserver to handle, so he puts a load balancer just in front of the webserver, and adds more Dell boxes running that same webserver software. Then, "he can have 2 or 3 or 50 webservers handling his traffic, but the database still talks to it over exactly the same HTTP API. This is the microservices approach: little software components talk to each other via APIs, and you can build and scale them independently," he says.

Microservices Examples From 3 Big Brands

An increasingly large amount of developers are choosing microservice architecture, including brands like Netflix, Facebook and Amazon. We also probed Rector, a former Google employee, about their take on microservice architecture: "[Building microservices] was 'the way we do things' at Google when I joined in 2006," Rector says.

He went on to add, that the architecture is most useful for the sort of cloud, mobile and SaaS applications that are reaching into other industries now. "It's relatively less common in telephony companies, for example, since most predate the cloud," he says. Here's a closer look at how some of these big brands are leveraging microservices architecture to deliver experiences at scale.

1. Netflix

Netflix, the movie and video streaming service, is one of many global brands that embraces microservices architecture. According to the Netflix Medium blog, the Netflix Content Platform Engineering team, "runs a number of business processes which are driven by asynchronous orchestration of tasks executing on microservices."

2. Uber

Uber's taxi-hailing app spans the globe, but it's not built as one, gigantic monolith. Uber adopted microservice architecture and explained that, "this [microservice] design pattern enforces the development of small services dedicated to specific, well-encapsulated domain areas. Each service can be written in its own language or framework, and can have its own database or lack thereof."

3. Groupon

Coupon giant Groupon transitioned from monolithic architecture to microservices architecture back in late 2012. They migrated their U.S. web traffic from a monolithic Ruby on Rails application to a new Node.js stack with substantial results. After the move, Groupon reported that "page loads are significantly faster across the site [and] our development teams can develop and ship features faster and with fewer dependencies on other teams."

# data[].attributes.attachment_link.title
What are Microservices and Can this Architecture Improve Your Application Development?

# data[].attributes.attachment_link.description
The software products we use every day for both business and leisure are going through a quiet transition behind the scenes. A decade ago, it was the norm to build software as one, large, unified product - as a monolith.

# data[].attributes.attachment_link.description
This tutorial shows you how to build a simple Dialogflow agent, walking you through the most important features of Dialogflow. You'll learn how to:

# data[].attributes.attachment_link.readable_text
Chatbots are taking the world by storm. Before we know it, we'll probably end up using them to find new funny cat videos.

The media have already dubbed it the 'new apps' (apps have completely transformed the way we perceive mobile technology, and now some think chatbots will do the same), and although we'll refrain from making such decisive statements, it's true that this technology has already left quite the mark. Some even fear the technology, arguing it might leave a lot of people without work.

Even though we're still only witnessing this technology's inception, we'd love to show you a couple of fantastic examples of chatbots, to help you understand how it's shaping the mobile user experience of the future. Yet before diving into some cool examples, let's take a brief glance at the way they work, to understand their power and to see how far we've come in terms of upgrading the digital interface.

The 'what' and the 'how' of chatbots

The 'only' thing that separates chatbots from 'regular' apps is its user interface. 'Regular' ones have a visual interface, comprised of buttons and icons. Chatbots have a chat interface, where the user literally chats with the app and operates it that way. As it is human nature to chat, and considering that not everyone has the time or can be bothered to learn an app inside-out, having such a simple and intuitive interface can greatly improve the overall mobile user experience.

We currently recognize two types of chatbots - those integrated into an app, and those using existing platforms, such as Facebook Messenger, or Kik. What's the difference? With some brands, for example H&M or CNN, you can converse through Kik, the messaging platform. For others, you'll need a dedicated app.

Why you should care about chatbots

You should care for one simple reason - the majority of your audience loves texting more than any other form of interaction through the mobile app.

People born in the '80s and younger actually prefer texting over talking, according to a recent OpenMarket report. It was also said that social media are the best channels for customer support. So it's safe to say that your target audience:

1) Loves to chat

2) Loves to use chat to get in touch with you as their favorite brand

3) Wants answers to common questions, and wants them fast

So let' take a look at a couple of fantastic examples how different brands, from different industries, used this new technology to create a better experience for their users:

eCommerce: Operator | iOS

Operator is an app designed to help people buy things. If you want to buy the perfect guitar for your seven-year-old, all you'd need to do is text Operator, and someone, somewhere, would begin researching to find the best deal for you. The chatbot plays the role of the middleman. It will read the user's message, analyze it and determine who of Operator's operators is best fit for the task.

This is also a good example of why chatbots are complementary to humans in the workplace.

Humans will still be the ones finding the deals, not the bot - the bot will be doing the simple, repetitive task. By getting repetitive tasks out of the way, the bot will give the worker more time, which can then be spent on more meaningful work.

Using chatbots for a virtual assistant-type app is a win-win situation. Users will have a better experience, as using the app will feel easier and more natural. Instead of filling out crude forms and ticking checkboxes to set up all the right parameters, all they need to do is chat with the app and tell it what they need.

They will also get better service, as chatbots are making sure the best operators are assigned for each task. And the app's developers can enjoy a growing app, full of happy customers.

Education: Duolingo | Android | iOS

The popular language learning app Duolingo has an interesting approach to chatbots. Duolingo wants to help its users practice a new language by conversing with, you guessed it - bots. Embracing the reality that conversation is considered among the best ways to learn a foreign language, Duolingo's users can talk to a bot for as long as they like, and the bot will correct them and suggest proper answers. Best of all, it can gauge the user's progress and increase its level, making it constantly challenging and interesting for the users. With chatbots, the app has gotten an almost human touch.

Medical: Babylon Health | Android | iOS

Babylon Health is a medical app which aims to put a 'doctor' in the palm of everyone's hands. It's similar to other medical apps in terms of what it's offering (remedy recommendations, possible side-effects, proper dosage, vitals monitoring), but its chat-oriented interface is what makes this app stand out.

Users can chat with the app about their symptoms 24/7, and get fast and accurate medical advice. Unlike simply typing a query on Google and hoping for the best, chatting with the app allows more tailored, personalized answers. The chatbot enables users to build off of a single question and provide specifics about their condition.

Obviously, an app can't replace a real doctor, so in the case you actually need one, you'll be able to talk to a general practitioner through the app, as well.

Wine & Dine: Reserve | Android | iOS

Reserve is a 'hospitality technology platform', an app allowing users to find restaurants, helps them book tables. The user can chat with the app through a platform resembling SMS - the entire reservation process feels like a human-to-human conversation. The app is not fully automated, though. It can handle simpler requests on its own, but if the user's requests are too much for it to handle (for example, if the bot can't find the type of restaurant or food you're looking for, it might transfer your request to a human for a more thorough list of recommendations) - the app will forward it to a member of the Reserve team.

Besides being able to find and book a table, the chatbot also allows users to always view the status of their requests and be able to easily make any changes. Obviously, this is something other apps of this type can do as well. However, none of them create the feeling as if the user is chatting to a friend who just happens to know every restaurant's manager, chef, entire staff, and has nothing better to do than sit there and help people make dinner reservations.

It's fast, it's simple, it's convenient and it really feels - custom tailored, personalized.

Chip is an intelligent money-saving app for iOS and Android which really takes the whole finance management thing to another level. There are a couple of features which really make the app stand out from others, like the automated transfer of funds into a savings account. It also happens to come with its own proprietary chatbot. By conversing with the app, users can quickly see how much they're spending, how much they're saving and how much they actually could be saving.

The messages the app sends to its users are mostly informal and friendly, filled with emojis and GIF images. Nick Ustinov (Chip co-founder) said the app has seen a 64 percent conversion from install to account creation, since they released a version with the chatbot. "That's double anything I've seen before using a traditional GUI."

Conclusion

Chatbots in apps are basically an upgrade to a mobile user interface, as they bring the most basic type of human interaction into the digital environment. A simpler, faster and more intuitive user interface results in an overall better user experience, which is one of the key factors for mobile growth.

However, not all apps, or app features, require this type of interaction. Specific functions within apps can be improved with chatbots because of their conversational and personalized attributes, and we will probably be seeing a lot more of this technology in the future.

# data[].attributes.attachment_link.title
5 examples of chatbots in apps taking us into the future

# data[].attributes.attachment_link.description
Chatbots are taking the world by storm. Before we know it, we'll probably end up using them to find new funny cat videos. The media have already dubbed it the 'new apps' (apps have completely transformed the way we perceive mobile technology, and now some think chatbots will do the same), and although we'll refrain from making such decisive statements, it's true that this technology has already left quite the mark.

# data[].attributes.attachment_link.readable_text
Natural Language Processing Summary

The field of study that focuses on the interactions between human language and computers is called Natural Language Processing, or NLP for short. It sits at the intersection of computer science, artificial intelligence, and computational linguistics ( Wikipedia).

"Natural Language Processing is a field that covers computer understanding and manipulation of human language, and it's ripe with possibilities for newsgathering," Anthony Pesce said in Natural Language Processing in the kitchen. "You usually hear about it in the context of analyzing large pools of legislation or other document sets, attempting to discover patterns or root out corruption."

What is Natural Language Processing?

NLP is a way for computers to analyze, understand, and derive meaning from human language in a smart and useful way. By utilizing NLP, developers can organize and structure knowledge to perform tasks such as automatic summarization, translation, named entity recognition, relationship extraction, sentiment analysis, speech recognition, and topic segmentation.

"Apart from common word processor operations that treat text like a mere sequence of symbols, NLP considers the hierarchical structure of language: several words make a phrase, several phrases make a sentence and, ultimately, sentences convey ideas," John Rehling, an NLP expert at Meltwater Group, said in How Natural Language Processing Helps Uncover Social Media Sentiment. "By analyzing language for its meaning, NLP systems have long filled useful roles, such as correcting grammar, converting speech to text and automatically translating between languages."

NLP is used to analyze text, allowing machines to understand how human's speak. This human-computer interaction enables real-world applications like automatic text summarization, sentiment analysis, topic extraction, named entity recognition, parts-of-speech tagging, relationship extraction, stemming, and more. NLP is commonly used for text mining, machine translation, and automated question answering.

NLP is characterized as a hard problem in computer science. Human language is rarely precise, or plainly spoken. To understand human language is to understand not only the words, but the concepts and how they're linked together to create meaning. Despite language being one of the easiest things for humans to learn, the ambiguity of language is what makes natural language processing a difficult problem for computers to master.

What Can Developers Use NLP Algorithms For?

NLP algorithms are typically based on machine learning algorithms. Instead of hand-coding large sets of rules, NLP can rely on machine learning to automatically learn these rules by analyzing a set of examples (i.e. a large corpus, like a book, down to a collection of sentences), and making a statical inference. In general, the more data analyzed, the more accurate the model will be.

Summarize blocks of text using Summarizer to extract the most important and central ideas while ignoring irrelevant information.

Create a chat bot using Parsey McParseface, a language parsing deep learning model made by Google that uses Point-of-Speech tagging.

Automatically generate keyword tags from content using AutoTag, which leverages LDA, a technique that discovers topics contained within a body of text.

Identify the type of entity extracted, such as it being a person, place, or organization using Named Entity Recognition.

Use Sentiment Analysis to identify the sentiment of a string of text, from very negative to neutral to very positive.

Reduce words to their root, or stem, using PorterStemmer, or break up text into tokens using Tokenizer.

Open Source NLP Libraries

These libraries provide the algorithmic building blocks of NLP in real-world applications. Algorithmia provides a free API endpoint for many of these algorithms, without ever having to setup or provision servers and infrastructure.

Apache OpenNLP: a machine learning toolkit that provides tokenizers, sentence segmentation, part-of-speech tagging, named entity extraction, chunking, parsing, coreference resolution, and more.

Natural Language Toolkit (NLTK): a Python library that provides modules for processing text, classifying, tokenizing, stemming, tagging, parsing, and more.

Standford NLP: a suite of NLP tools that provide part-of-speech tagging, the named entity recognizer, coreference resolutionsystem, sentiment analysis, and more.

MALLET: a Java package that provides Latent Dirichlet Allocation, document classification, clustering, topic modeling, information extraction, and more.

A Few NLP Examples

Use Summarizer to automatically summarize a block of text, exacting topic sentences, and ignoring the rest.

Generate keyword topic tags from a document using LDA (Latent Dirichlet Allocation), which determines the most relevant words from a document. This algorithm is at the heart of the Auto-Tag and Auto-Tag URL microservices.

Sentiment Analysis, based on StanfordNLP, can be used to identify the feeling, opinion, or belief of a statement, from very negative, to neutral, to very positive. Often, developers with use an algorithm to identify the sentiment of a term in a sentence, or use sentiment analysis to analyze social media.

NLP algorithms can be extremely helpful for web developers, providing them with the turnkey tools needed to create advanced applications, and prototypes.

Example Natural Language Processing Use Cases

NLP algorithms are typically based on machine learning algorithms. Instead of hand-coding large sets of rules, NLP can rely on machine learning to automatically learn these rules by analyzing a set of examples (i.e. a large corpus, like a book, down to a collection of sentences), and making a statical inference. In general, the more data analyzed, the more accurate the model will be.

Social media analysis is a great example of NLP use. Brands track conversations online to understand what customers are saying, and glean insight into user behavior.

"One of the most compelling ways NLP offers valuable intelligence is by tracking sentiment - the tone of a written message (tweet, Facebook update, etc.) - and tag that text as positive, negative or neutral," Rehling said.
Build your own social media monitoring tool

1. Start by using the algorithm Retrieve Tweets With Keyword to capture all mentions of your brand name on Twitter. In our case, we search for mentions of Algorithmia.

2. Then, pipe the results into the Sentiment Analysis algorithm, which will assign a sentiment rating from 0-4 for each string (Tweet).

Similarly, Facebook uses NLP to track trending topics and popular hashtags.

"Hashtags and topics are two different ways of grouping and participating in conversations," Chris Struhar, a software engineer on News Feed, said in How Facebook Built Trending Topics With Natural Language Processing. "So don't think Facebook won't recognize a string as a topic without a hashtag in front of it. Rather, it's all about NLP: natural language processing. Ain't nothing natural about a hashtag, so Facebook instead parses strings and figures out which strings are referring to nodes - objects in the network. We look at the text, and we try to understand what that was about."

It's not just social media that can use NLP to it's benefit. Publishers are hoping to use NLP to improve the quality of their online communities by leveraging technology to "auto-filter the offensive comments on news sites to save moderators from what can be an 'exhausting process'," Francis Tseng said in Prototype winner using 'natural language processing' to solve journalism's commenting problem.

Other practical uses of NLP include monitoring for malicious digital attacks, such as phishing, or detecting when somebody is lying.
Use NLP to build your own RSS reader
You can build a machine learning RSS reader in less than 30-minutes using the follow algorithms:

1. ScrapeRSS to grab the title and content from an RSS feed.

2. Html2Text to keep the important text, but strip all the HTML from the document.

3. AutoTag uses Latent Dirichlet Allocation to identify relevant keywords from the text.

4. Sentiment Analysis is then used to identify if the article is positive, negative, or neutral.

5. Summarizer is finally used to identify the key sentences.

Recommended NLP Books for Beginners

Speech and Language Processing: "The first of its kind to thoroughly cover language technology - at all levels and with all modern technologies - this book takes an empirical approach to the subject, based on applying statistical and other machine-learning algorithms to large corporations."

Foundations of Statistical Natural Language Processing: "This foundational text is the first comprehensive introduction to statistical natural language processing (NLP) to appear. The book contains all the theory and algorithms needed for building NLP tools. It provides broad but rigorous coverage of mathematical and linguistic foundations, as well as detailed discussion of statistical methods, allowing students and researchers to construct their own implementations. The book covers collocation finding, word sense disambiguation, probabilistic parsing, information retrieval, and other applications."

Handbook of Natural Language Processing: "The Second Edition presents practical tools and techniques for implementing natural language processing in computer systems. Along with removing outdated material, this edition updates every chapter and expands the content to include emerging areas, such as sentiment analysis."

Statistical Language Learning (Language, Speech, and Communication): "Eugene Charniak breaks new ground in artificial intelligenceresearch by presenting statistical language processing from an artificial intelligence point of view in a text for researchers and scientists with a traditional computer science background."

NLP Tutorials

Natural Language Processing Tutorial: "We will go from tokenization to feature extraction to creating a model using a machine learning algorithm. You can get the source of the post from github."

Basic Natural Language Processing: "In this tutorial competition, we dig a little "deeper" into sentiment analysis. People express their emotions in language that is often obscured by sarcasm, ambiguity, and plays on words, all of which could be very misleading for both humans and computers."

An NLP tutorial with Roger Ebert: "Natural Language Processing is the process of extracting information from text and speech. In this post, we walk through different approaches for automatically extracting information from text-keyword-based, statistical, machine learning-to explain why many organizations are now moving towards the more sophisticated machine-learning approaches to managing text data."

If you're interested in learning more, this free introductory course from Stanford University will help you will learn the fundamentals of natural language processing, and how you can use it to solve practical problems.

Once you've gotten the fundamentals down, apply what you've learned using Python and NLTK, the most popular framework for Python NLP.

NLP Videos

NLP Courses

Udemy's Introduction to Natural Language Processing
"This course introduces Natural Language Processing through the use of python and the Natural Language Tool Kit. Through a practical approach, you'll get hands on experience working with and analyzing text. As a student of this course, you'll get updates for free, which include lecture revisions, new code examples, and new data projects."

Natural language processing (Wikipedia): "Natural language processing (NLP) is a field of computer science, artificial intelligence, and computational linguistics concerned with the interactions between computers and human (natural) languages. In 1950, Alan Turing published an article titled 'Computing Machinery and Intelligence' which proposed what is now called the Turing test as a criterion of intelligence. Starting in the late 1980s, however, there was a revolution in NLP with the introduction of machine learning algorithms for language processing."

Outline of natural language processing (Wikipedia): "The following outline is provided as an overview of and topical guide to natural language processing: Natural language processing - computer activity in which computers are entailed to analyze, understand, alter, or generate natural language."

Apache OpenNLP: "The Apache OpenNLP library is a machine learning based toolkit for the processing of natural language text."

Natural Language Toolkit: "NLTK is a leading platform for building Python programs to work with human language data. It provides easy-to-use interfaces to over 50 corpora and lexical resources such as WordNet, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning, wrappers for industrial-strength NLP libraries, and an active discussion forum. Natural Language Processing with Python provides a practical introduction to programming for language processing."

Six Natural Language Processing Algorithms for Web Developers

A curated list of speech and natural language processing resources

NLP research group at google

General Introduction to NLP

Natural language processing: an introduction

Stanford CS 224D Video: Deep Learning for Natural Language Processing

CS 388: Natural Language Processing

COMS W4705: Natural Language Processing

CS 674: Natural Language Processing

CS918 Natural Language Processing

r/LanguageTechnology

Everything You Need to Know about Natural Language Processing

# data[].attributes.attachment_link.title
Introduction to Natural Language Processing (NLP) - Algorithmia Blog

# data[].attributes.attachment_link.description
The field of study that focuses on the interactions between human language and computers is called Natural Language Processing, or NLP for short. It sits at the intersection of computer science, artificial intelligence, and computational linguistics ( Wikipedia).

# data[].attributes.attachment_link.title
Using Natural Language in Software Development

# data[].attributes.attachment_link.description
This paper presents a new approach to conceptual modeling based on natural language analysis.

# data[].attributes.attachment_link.title
Software - The Stanford Natural Language Processing Group

# data[].attributes.attachment_link.description
The Stanford NLP Group makes some of our Natural Language Processing software available to everyone! We provide statistical NLP, deep learning NLP, and rule-based NLP tools for major computational linguistics problems, which can be incorporated into applications with human language technology needs. These packages are widely used in industry, academia, and government.

# data[].attributes.attachment_link.readable_text
Natural Language Processing, or NLP for short, is broadly defined as the automatic manipulation of natural language, like speech and text, by software.

The study of natural language processing has been around for more than 50 years and grew out of the field of linguistics with the rise of computers.

In this post, you will discover what natural language processing is and why it is so important.

After reading this post, you will know:

What natural language is and how it is different from other types of data.

What makes working with natural language so challenging.

Where the field of NLP came from and how it is defined by modern practitioners.

Let's get started.

Natural Language

Natural language refers to the way we, humans, communicate with each other.

Namely, speech and text.

We are surrounded by text.

Think about how much text you see each day:

Signs

Menus

Email

SMS

Web Pages

and so much more...

The list is endless.

Now think about speech.

We may speak to each other, as a species, more than we write. It may even be easier to learn to speak than to write.

Voice and text are how we communicate with each other.

Given the importance of this type of data, we must have methods to understand and reason about natural language, just like we do for other types of data.

Need help with Deep Learning for Text Data?

Take my free 7-day email crash course now (with code).

Click to sign-up and also get a free PDF Ebook version of the course.
Start Your FREE Crash-Course Now

Challenge of Natural Language

Working with natural language data is not solved.

It has been studied for half a century, and it is really hard.

It is hard from the standpoint of the child, who must spend many years acquiring a language ... it is hard for the adult language learner, it is hard for the scientist who attempts to model the relevant phenomena, and it is hard for the engineer who attempts to build systems that deal with natural language input or output. These tasks are so hard that Turing could rightly make fluent conversation in natural language the centerpiece of his test for intelligence.

- Page 248, Mathematical Linguistics, 2010.

Natural language is primarily hard because it is messy. There are few rules.

And yet we can easily understand each other most of the time.

Human language is highly ambiguous ... It is also ever changing and evolving. People are great at producing language and understanding language, and are capable of expressing, perceiving, and interpreting very elaborate and nuanced meanings. At the same time, while we humans are great users of language, we are also very poor at formally understanding and describing the rules that govern language.

- Page 1, Neural Network Methods in Natural Language Processing, 2017.

From Linguistics to Natural Language Processing

Linguistics

Linguistics is the scientific study of language, including its grammar, semantics, and phonetics.

Classical linguistics involved devising and evaluating rules of language. Great progress was made on formal methods for syntax and semantics, but for the most part, the interesting problems in natural language understanding resist clean mathematical formalisms.

Broadly, a linguist is anyone who studies language, but perhaps more colloquially, a self-defining linguist may be more focused on being out in the field.

Mathematics is the tool of science. Mathematicians working on natural language may refer to their study as mathematical linguistics, focusing exclusively on the use of discrete mathematical formalisms and theory for natural language (e.g. formal languages and automata theory).

Computational Linguistics

Computational linguistics is the modern study of linguistics using the tools of computer science. Yesterday's linguistics may be today's computational linguist as the use of computational tools and thinking has overtaken most fields of study.

Computational linguistics is the study of computer systems for understanding and generating natural language. ... One natural function for computational linguistics would be the testing of grammars proposed by theoretical linguists.

- Pages 4-5, Computational Linguistics: An Introduction, 1986.

Large data and fast computers mean that new and different things can be discovered from large datasets of text by writing and running software.

In the 1990s, statistical methods and statistical machine learning began to and eventually replaced the classical top-down rule-based approaches to language, primarily because of their better results, speed, and robustness. The statistical approach to studying natural language now dominates the field; it may define the field.

Data-Drive methods for natural language processing have now become so popular that they must be considered mainstream approaches to computational linguistics. ... A strong contributing factor to this development is undoubtedly the increase amount of available electronically stored data to which these methods can be applied; another factor might be a certain disenchantment with approaches relying exclusively on hand-crafted rules, due to their observed brittleness.

- Page 358, The Oxford Handbook of Computational Linguistics, 2005.

The statistical approach to natural language is not limited to statistics per-se, but also to advanced inference methods like those used in applied machine learning.

... understanding natural language require large amounts of knowledge about morphology, syntax, semantics and pragmatics as well as general knowledge about the world. Acquiring and encoding all of this knowledge is one of the fundamental impediments to developing effective and robust language systems. Like the statistical methods ... machine learning methods off the promise of automatic the acquisition of this knowledge from annotated or unannotated language corpora.

- Page 377, The Oxford Handbook of Computational Linguistics, 2005.

Statistical Natural Language Processing

Computational linguistics also became known by the name of natural language process, or NLP, to reflect the more engineer-based or empirical approach of the statistical methods.

The statistical dominance of the field also often leads to NLP being described as Statistical Natural Language Processing, perhaps to distance it from the classical computational linguistics methods.

I view computational linguistics as having both a scientific and an engineering side. The engineering side of computational linguistics, often called natural language processing (NLP), is largely concerned with building computational tools that do useful things with language, e.g., machine translation, summarization, question-answering, etc. Like any engineering discipline, natural language processing draws on a variety of different scientific disciplines.

- How the statistical revolution changes (computational) linguistics, 2009.

Linguistics is a large topic of study, and, although the statistical approach to NLP has shown great success in some areas, there is still room and great benefit from the classical top-down methods.

Roughly speaking, statistical NLP associates probabilities with the alternatives encountered in the course of analyzing an utterance or a text and accepts the most probable outcome as the correct one. ... Not surprisingly, words that name phenomena that are closely related in the world, or our perception of it, frequently occur close to one another so that crisp facts about the world are reflected in somewhat fuzzier facts about texts. There is much room for debate in this view.

- Page xix, The Oxford Handbook of Computational Linguistics, 2005.

Natural Language Processing

As machine learning practitioners interested in working with text data, we are concerned with the tools and methods from the field of Natural Language Processing.

We have seen the path from linguistics to NLP in the previous section. Now, let's take a look at how modern researchers and practitioners define what NLP is all about.

In perhaps one of the more widely textbooks written by top researchers in the field, they refer to the subject as " linguistic science," permitting discussion of both classical linguistics and modern statistical methods.

The aim of a linguistic science is to be able to characterize and explain the multitude of linguistic observations circling around us, in conversations, writing, and other media. Part of that has to do with the cognitive size of how humans acquire, produce and understand language, part of it has to do with understanding the relationship between linguistic utterances and the world, and part of it has to do with understand the linguistic structures by which language communicates.

- Page 3, Foundations of Statistical Natural Language Processing, 1999.

They go on to focus on inference through the use of statistical methods in natural language processing.

Statistical NLP aims to do statistical inference for the field of natural language. Statistical inference in general consists of taking some data (generated in accordance with some unknown probability distribution) and then making some inference about this distribution.

- Page 191, Foundations of Statistical Natural Language Processing, 1999.

In their text on applied natural language processing, the authors and contributors to the popular NLTK Python library for NLP describe the field broadly as using computers to work with natural language data.

We will take Natural Language Processing - or NLP for short -in a wide sense to cover any kind of computer manipulation of natural language. At one extreme, it could be as simple as counting word frequencies to compare different writing styles. At the other extreme, NLP involves "understanding" complete human utterances, at least to the extent of being able to give useful responses to them.

- Page ix, Natural Language Processing with Python, 2009.

Statistical NLP has turned another corner and is now strongly focused on the use of deep learning neural networks to both perform inference on specific tasks and for developing robust end-to-end systems.

In one of the first textbooks dedicated to this emerging topic, Yoav Goldberg succinctly defines NLP as automatic methods that take natural language as input or produce natural language as output.

Natural language processing (NLP) is a collective term referring to automatic computational processing of human languages. This includes both algorithms that take human-produced text as input, and algorithms that produce natural looking text as outputs.

- Page xvii, Neural Network Methods in Natural Language Processing, 2017.

Further Reading

This section provides more resources on the topic if you are looking go deeper.

Books

Wikipedia

Summary

In this post, you discovered what natural language processing is why it is so important.

Specifically, you learned:

What natural language is and how it is different from other types of data.

What makes working with natural language so challenging.

Where the field of NLP came from and how it is defined by modern practitioners.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Develop Deep Learning models for Text Data Today!

Develop Your Own Text models in Minutes

...with just a few lines of python code

Discover how in my new Ebook:
Deep Learning for Natural Language Processing

It provides self-study tutorials on topics like:
Bag-of-Words, Word Embedding, Language Models, Caption Generation, Text Translation and much more...

Finally Bring Deep Learning to your Natural Language Processing Projects

Skip the Academics. Just Results.

Click to learn more.

# data[].attributes.attachment_link.description
Natural Language Processing, or NLP for short, is broadly defined as the automatic manipulation of natural language, like speech and text, by software. The study of natural language processing has been around for more than 50 years and grew out of the field of linguistics with the rise of computers.

# data[].attributes.attachment_link.title
Image Recognition Libraries and Projects List by Language

# data[].attributes.attachment_link.title
Best Image Recognition Software in 2018 | G2 Crowd

# data[].attributes.attachment_link.description
Find the best Image Recognition Software using real-time, up-to-date data from over 84 verified user reviews. Read unbiased insights, compare features & see pricing for 41 solutions. Request demos & free trials to discover the right product for your business.

# data[].attributes.attachment_link.title
List of 10+ Face Detection / Recognition APIs, libraries, and software - KongHQ

# data[].attributes.attachment_link.description
Mashape presents a list of 10+ Face Detection / Face Recognition APIs, libraries, and software that you can use for your applications.

# data[].attributes.attachment_link.readable_text
Raspberry Pi Facial Recognition using AWS Rekognition and Pi-Timolo

Pi-detector is used with Pi-Timolo to search motion generated images for face matches by leveraging AWS Rekognition. In its current state, matches are wrote to event.log. With some additional creativity and work, you could send out a notification or allow/deny access to a room with minimal changes. The install script will place the appropriate files in /etc/rc.loal to start on boot.

Raspberry Pi (Tested with Rpi 3)

AWS Rekognition Access (Free tier options available)

As an alternative, this set of scripts can be modified to watch any directory that contains images. For example, if you collect still images from another camera and save them to disk, you can alter the image path to run facial recognition against any new photo that is created.

Before installing, it is best to get up and running on AWS. For my project, I am using the AWS Free Tier service. Using this allows you 5000 API calls per month, which is good enough for this project. Login to your console and create a new IAM user that has Administrative rights to Rekognition.

View of the user's groups.

If you need help with this step, here is the reference. The thing you need to do is copy down your aws_access_key_id and aws_access_key_secret. To obtain this information, click on the 'security credentials' tab (from the IAM users page as seen in the images above). If this is your first time, you may have to generate a new secret. You will be asked this information once you start the install script.
Setup a Raspberry Pi with Raspbian Jessie
Once your user has access to Rekognition, make sure you set your region to US-East because it is one of the few that offer Rekognition services. To change regions, go to EC2, Rekognition, or any other service and look in the upper right hand corner.

SSH into your Raspberry pi (or connect it to a monitor and login using pi as the username and raspberry as the password). Don't forget to change the below IP address to your pi's IP. If you need help finding it on the network use nmap (nmap -sn 192.168.1.0/24)

Clone this repo and install:

git clone https://github.com/af001/pi-detector.git cd pi-detector/scripts sudo chmod +x install.sh sudo ./install.sh

During the install, you will be prompted for your aws credentials that you set up earlier. When asked, enter your AWS Secret Key ID, AWS Secret Access Key, and set the region to us-east-1 (adjust to match the region you chose when you set up AWS Rekognition earlier). An example output would look something similar to the image below:

First, you need to create a new collection on AWS Rekognition. Creating a 'home' collection would look like:

Next, add your images to the pi-detector/faces folder. The more images of a person the better results you will get for detection. I would recommend several different poses in different lighting.

I found the best results by taking a photo in the same area that the camera will be placed, and by using the picam. If you want to do this, I created a small python script to take a photo with a 10 second delay and then puts it into the pi-detector/faces folder. To use it:

Once complete, you can go back and rename the file and repeat the steps above to add your images to AWS Rekognition. Once you create a new collection, or add a new image, two reference files will be created as a future reference. These are useful if you plan on deleting images or collections in the future.

At this point, the setup is ready to go. You can setup Wi-Fi on your Rpi and place the camera where you want in your home. Once you plug in the Rpi, it should start working with no additional work needed from the user. To check your logs, just ssh into the Rpi and check the event.log folder for a reference to your detections.

To delete a face from your collection, use the following:

If you need to find the image id or a collection name, reference your faces.txt and collections.txt files.

To remove a collection:

Note that the above will also delete all the faces you have stored in AWS.

The last script is facematch.py. If you have images updated and just want to test static photos against the faces you have stored on AWS, do the following:

The results will be printed to screen, to include the percentage of similarity and confidence.

Next, I will be working on adding the servo to this project to open a door lock. Once I get it up and going, I will update this post. Also, I will be adding a video of the entire install and setup shortly.

# data[].attributes.attachment_link.description
Raspberry Pi Facial Recognition using AWS Rekognition and Pi-Timolo Pi-detector is used with Pi-Timolo to search motion generated images for face matches by leveraging AWS Rekognition. In its current state, matches are wrote to event.log. With some additional creativity and work, you could send out a notification or allow/deny access to a room with minimal changes.

# data[].attributes.attachment_link.readable_text
Today's post is after reading the below book on Microservices and various Service fabric article readings and research. I am planning to break it down into series of blog post explaining Service Fabric and things to learn for developers or architects in coming days hopefully, wish me good luck ;).. Comments welcome.

Contents and Samples of #Recommend bob's book is here #microservices #servicefabric #azure https://t.co/eOaXbnpCB6 pic.twitter.com/hM8da8glQa

- Maheshk (@MahesKBlr) January 9, 2016

What is this Microservices and why there is huge fuzz in recent times:-

Microservices are nothing but fine grained SOA offerings. We can say more politely "it does one thing but very well" means without affecting others or waiting something to trigger. It gives lot of freedom for developers more importantly to decide how to use, design, language to choose, platform to run, deploy, scale etc. It decouples from the main deliverable but revolves in satellite mode support with thick coupling.

Let us see one by one:-

1) It help us to develop, deliver as a smaller component services importantly in an agile way. Each of these services can be owned by a smaller team can be fed with 2 Pizza's team which they call in Amazon (read about 2PT's) which I picked from this fantastic book.

2) It gives us the freedom to develop and deploy services independently without waiting for the complete release say quarterly release or half yearly release cycle as in monolithic world. The consumer can start seeing the enhancements or features rolled out now and then and enjoy greatly.

3) These Microservices though deployed separately, but it has to be orchestrated together in one way or another to work closely for a bigger applications. It is designed to run as a unique process or service in silos bundled including separate DB to maintain its state values. It communicate with each other services over Web API/Broker messages way of signalling and syncing. By going this way, we make the team into small units owning the end-to end service responsibility focusing and pushing without waiting for a big release day.

4) DevOps role kind of guys comes in here to develop, configure, release, maintain and also monitor these services exposed as an API endpoint or stateless REST API's or HTTP/XML/TCP/JSON and of course even binary format too. Typical example would be, heavy trafficked eCommerce site like Amazon, the recommendation shown at the bottom of the page would come from one such microservice, payment gateway code logic on another microservice, or themes or catalogue.. so who knows there could be 25+ such microservices supporting the complete portal. These kind of service separation allows the developer guys to evolve quickly and also fix their bugs/changes in real time by updating one after the other. Live example is, amazon.com relies on more than 150+ API's to render its any of their page.

5) These services can be written in any programming language or using any framework hosted in any platform too. It gives us the choice to decide and roll out quickly with our known languages.

6) It gives us the flexibility to test out these individual services individually for scaling scenarios to sustain black friday sale days kind of sudden burst.

7) As I said earlier, these services do one thing so it has to do well. So having this in mind, developers design them to live and serve for longer time by having newer version side by side. In case of an issue in deployment, we can always redirect to older or newer version in agile way. Most of the steps are carried out through scripts or portals GUI for config changes to act quickly

8) Next important benefit is " resilience". A typical monolithic application would be deployed in big fat server whereas these microservices would spread across servers or even between different data center also. If one such service goes down, then the request would get served from other available copy to continue the business as usual. It also allow us to "scale out" efficiently with duplicate copies of those critical core services by deploying across servers/VMs/containers. There are open source tools which comes handy to perform these scaling operation. In monolithic design, we can maximum scale up to support the additional load but cannot live in separation- tightly coupled in same basket.

9) List of companies known for using Microservice as an early adaptor are Amazon, Netflix, Twitter, Bluemix, Souldcloud, Google, Azure. In recent times, many many big websites and application has already moved from monolithic to Microservices architecture to reap it benefits. Recently, this area is hotter than Big Data or IoT trend and also players like Google Kubernetes, Mesos/Marathon, Docker Swarm, Amazon ECS, CoreOS Fleet or Rancher, Azure Container Service, Service Fabric etc are there to fill in one or the other form. The Main objective of the Service Fabric is to reduce the complexities in building complex applications using Microservice approach. Definitely Containers and actor based programming model are the examples but as said earlier, this area is going to have more update this year.

10) Microservices are usually designed to function as stateless because stateful is tricky & difficult to design for storing its state maintained. But Service Fabric allows us to build such stateless/stateful service without much effort. Service Fabric is a PaaS system for building highly scalable, distributed applications composed from microservices. It enables an > application runtime, > patching and innovative rolling upgrade > health-model support > all while maintaining local state replication. I will try to cover the rest in next part in detail with hands on. Here is the bunch of links to think through.
Microservices:

1. Why a microservices approach to building applications? https://azure.microsoft.com/en-us/documentation/articles/service-fabric-overview-microservices/

2. Virtual Academy: Exploring Microservice Architecture : https://azure.microsoft.com/en-us/blog/virtual-academy-exploring-microservice-architecture/

3. Provision and deploy microservices predictably in Azure : https://azure.microsoft.com/en-us/documentation/articles/app-service-deploy-complex-application-predictably/

4. http://www.microservicesweekly.com/ - * highly recommended *

General resources:

1. Microservices : http://martinfowler.com/articles/microservices.html

2. Microservices Are Not a free lunch! : http://contino.co.uk/microservices-not-a-free-lunch/

3. Services, Microservices, Nanoservices - oh my! : http://arnon.me/2014/03/services-microservices-nanoservices/

4. Micro Service Architecture : https://yobriefca.se/blog/2013/04/29/micro-service-architecture/

5. Microservice Design Patterns : https://www.voxxed.com/blog/2015/04/microservice-design-patterns/

6. Microservices: Decomposing Applications for Deployability and Scalability : http://www.infoq.com/articles/microservices-intro

Service Fabric -specific:

1. Microsoft Azure Service Fabric Architecture : https://channel9.msdn.com/Events/Ignite/2015/BRK3717

2. Building Resilient, Scalable Services with Microsoft Azure Service Fabric : https://channel9.msdn.com/Events/Ignite/2015/BRK3730

3. Deploying and Managing Services with Microsoft Azure Service Fabric : https://channel9.msdn.com/Events/Ignite/2015/BRK3478

4. Service Orchestration with Microsoft Azure Service Fabric : https://channel9.msdn.com/Events/Ignite/2015/BRK3485

5. Microsoft Azure Service Fabric Actors: The Director's Cut : https://channel9.msdn.com/Events/Ignite/2015/BRK3476

6. Microservices: An application revolution powered by the cloud:- https://azure.microsoft.com/en-us/blog/microservices-an-application-revolution-powered-by-the-cloud/

7. Guide to converting Web and Worker Roles to Service Fabric stateless services - https://azure.microsoft.com/en-us/documentation/articles/service-fabric-cloud-services-migration-worker-role-stateless-service
Learn about the differences between Cloud Services and Service Fabric before migrating applications. -https://azure.microsoft.com/en-us/documentation/articles/service-fabric-cloud-services-migration-differences/

btw, As I am writing this, I do see Service Fabric going live or GA (general availability). Worth to check //BUILD2016 events and news about ths same topic for latest happenings https://channel9.msdn.com/Events/Build/2016
happy learning !

# data[].attributes.attachment_link.title
What are Microservices for developers - Part 1

# data[].attributes.attachment_link.description
Today's post is after reading the below book on Microservices and various Service fabric article readings and research. I am planning to break it down into series of blog post explaining Service Fabric and things to learn for developers or architects in coming days hopefully, wish me good luck ;)..

# data[].attributes.attachment_link.readable_text
Date: July 12th, 2018 Kubernetes Version: v1.11.0 📝 is a sidebar ⚠️ is a warning

Kubernetes is a container orchestration system. You can think of a container as an application and multiple containers as a pod. The orchestration part means you tell Kubernetes what containers you want running and it will take care of actually running the containers in your cluster, routing traffic to the correct pods, and many other features.

Kubernetes provides a schema for your pod definitions. This means you fill in a well documented template and give it to the Kubernetes cluster. Kubernetes then figures out what will run where, runs your pods and configures the cluster network. If a pod crashes, Kubernetes will notice the system is not in the correct state anymore. Kubernetes will act to return the system to the defined state.

Let's look at an example: A blogging platform called devtoo.com on a Kubernetes cluster.

Components

The first step is to figure out the components of devtoo.com. Let's say these are all the components necessary:

1. A web server that accepts HTTP traffic from the internet. Examples of web servers include nginx and apache

2. An application server that loads the rails app into memory and serves requests. This would be the rails application that powers devtoo.com.

3. A database to store all of our awesome posts. Postgres, mysql and MongoDB are all database examples.

4. A cache to bypass the application and database and immediately return a result. Examples of caches include redis and memcached.

The end goal

The next step is to figure out what the final system should look like. Kubernetes gives you a lot of choice here. The components could each run in their own pod or they could all be put into one pod. I like to start at the simplest place and then fix the solution if it sucks. To me, that means each component will be run in its own pod. A typical web request will enter the system and hit the web server. The web server will ask the cache if it has a result for that endpoint. If it does, the result is returned immediately. If it does not, the request is passed on to the application server. The application server is configured to talk to the database and generate dynamic content which gets sent back to the web browser.

Defining the system

Kubernetes maps services to pods. There will be one service for each pod. This will allow you to reference other pods with DNS.

The service defines a selector, app: web. The service will route traffic to any pod that matches that selector. If you look at the pod definition you will see that there is an app: web label defined on the pod. That means traffic comes into the service on port 80 and gets sent to the nginx pod on the targetPort, also 80 in this case. The targetPort and containerPort must match.

Here, you use your magic wand and produce an nginx config that is embedded in the nginx image that sends traffic to the cache and then if there is no result to the app server.

Here is the cache definition:

And the database definition:

Those are all of the dependencies that were considered for this deployment of devtoo.com. Next the application itself must be configured. Rails can use an environment variable to connect to a database. You could define that in the pod YAML like this:

⚠️This is super insecure! Kubernetes has much better ways to do this but I'm omitting them to keep the scope of this post "small".

The last piece needed is an ingress point, a place where traffic can enter the cluster from the outside world.

📝I'm glossing over IngressControllers because, while required, they are an implementation detail to be ignored at this level of understanding.

This says that any traffic received at this ingress point will be sent to the service with a name of web on port 80.

Now your cluster is set up, let's trace a packet to get this blogpost. You enter http://devtoo.com/chuck_ha/this-post into your browser. http://devtoo.com resolves in DNS to some IP address which is a load balancer in front of your kubernetes cluster. The load balancer sends the traffic to your ingress point. Since there is only one service on the ingress, the traffic is then sent to the web service which is mapped to the nginx pod. The nginx container inspects the packet and sends it to the cache service which is mapped to the redis pod. The redis pod has never seen this URL before so execution continues from nginx. The request is sent to the application server where this page is generated, cached and returned to your web browser.

Then you click on the 🦄 button!

📝 A list of things I skipped so you could focus on the meat and not get lost in the details:

Did you find this post useful? Show some love!

dev.to is where software developers stay in the loop and avoid career stagnation. Signing up (for free!) is the first step.

# data[].attributes.attachment_link.title
Understanding Kubernetes through a concrete example

# data[].attributes.attachment_link.description
Date: July 12th, 2018 Kubernetes Version: v1.11.0 📝 is a sidebar ⚠️ is a warning Kubernetes is a container orchestration system. You can think of a container as an application and multiple containers as a pod.

# data[].attributes.attachment_link.readable_text
The microservices architecture is a variant of service oriented architecture. It develops a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API.
'Microservices' is a compound word made of 'micro' and 'services'. As the name suggests, microservices are the small modules that provide some functionality to a system. These modules can be anything that is designed to serve some specific function. These services can be independent or interrelated with each other, based on some contract.

The main function of microservices is to provide isolation between services - a separation of services from servers and the ability to run them independently, with the interaction between them based on a specific requirement. To achieve this isolation, we use containerisation, which will be discussed later. The idea behind choosing microservices is to avoid correlated failure in a system where there is a dependency between services. When running all microservices inside the same process, all services will be killed if the process is restarted. By running each service in its own process, only one service is killed if that process is restarted, but restarting the server will kill all services. By running each service on its own server, it's easier to maintain these isolated services, though there is a cost associated with this option.
How microservices are defined
The microservices architecture develops a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API.

Microservices is a variant of the service-oriented architecture (SOA) architectural style. A SOA is a style of software design in which services are provided to the other components by application components, through a communication protocol over a network that structures an application as a collection of loosely coupled services. In the microservices architecture, services should be fine-grained and the protocols should be lightweight. The benefit of breaking down an application into different smaller services is that it improves modularity and makes the application easier to understand, develop and test. It also parallelises development by enabling small autonomous teams to develop, deploy and scale their respective services independently.

These services are independently deployable and scalable. Each service also provides a kind of contract allowing for different services to be written in different programming languages. They can also be managed by different teams.
The architecture of microservices
Microservices follows the service-oriented architecture in which the services are independent of users, products and technologies. This architecture allows one to build applications as suites of services that can be used by other services. This architecture is in contrast to the monolithic architecture, where the services are built as a single unit comprising a client-side user interface, databases and server-side applications in a single frame - all dependent on one another. The failure of one can bring down the whole system.

The microservices architecture mainly consists of the client-side user interface, databases and server-side applications as different services that are related in some way to each other but are not dependent on each other. Each layer is independent of the other, which in turn leads to easy maintenance. The architecture is represented in Figure 2.

This architecture is a form or system that is built by plugging together components, somewhat like in a real world composition where a component is a unit of software that is independently replaceable and upgradeable. These microservices are easily deployable and integrated into one another. This gives rise to the possibility of continuous integration and continuous deployment.
What's so good about microservices?
With the advances in software architecture, microservices have emerged as a different platform compared to other software architecture. Microservices are easily scalable and are not limited to a language; so you are free to choose any language for the services. The services are loosely coupled, which in turn results in ease of maintenance and flexibility, as well as reduced time in debugging and deployment.
Microservices with Docker and Kubernetes
Docker is a software technology that provides containers, which are a computer virtualisation method in which the kernel of an operating system allows the existence of multiple isolated user-space instances, instead of just one. Everything required to make a piece of software run is packaged into isolated containers. With microservices, containers play the same role of providing virtual environments to different processes that are running, being deployed and undergoing testing, independently.

Docker is a bit like a virtual machine, but rather than creating a whole virtual operating system, Docker allows applications to use the same kernel as the system that it's running on and only requires applications to be shipped with things not already running on the host computer. The main idea behind using Docker is to eliminate the 'works on my machine' type of problems that occur when collaborating on code with co-workers. Docker doesn't have to install and configure complex databases nor worry about switching between incompatible language toolchain versions. When an app is dockerised, that complexity is pushed into containers that are easily built, shared and run. It is a tool that is designed to benefit both developers and systems administrators.
How well does Kubernetes go with Docker?
Before starting the discussion on Kubernetes, we must first understand orchestration, which is to arrange various components so that they achieve a desired result. It also means the process of integrating two or more applications and/or services together to automate a process, or synchronise data in real-time.

The intermediate path connecting two or more services is done by orchestration, which refers to the automated arrangement, coordination and management of software containers. So what does Kubernetes do then? Kubernetes is an open source platform for automating deployments, scaling and operations of application containers across clusters of hosts, providing container-centric infrastructure. Orchestration is an idea whereas Kubernetes implements that idea. It is a tool for orchestration. It deploys containers inside a cluster. It is a helper tool that can be used to manage a cluster of containers and treat all servers as a single unit. These containers are provided by Docker.

The best example of Kubernetes is the Pokémon Go App, which runs on a virtual environment of Google Cloud, in a separate container for each user. Kubernetes uses a different set-up for each OS. So if you want a tool that will overcome Docker's limitations, you should go with Kubernetes.

To conclude, we may say that microservices is growing very fast, the reason being its features of independence and isolation which give it the power to easily run, test and be deployed. This is just a small summary of microservices, about which there is a lot more to learn.

# data[].attributes.attachment_link.title
A glimpse of microservices with Docker and Kubernetes - Open Source For You

# data[].attributes.attachment_link.description
The microservices architecture is a variant of service oriented architecture. It develops a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API. 'Microservices' is a compound word made of 'micro' and 'services'.

# data[].attributes.attachment_link.readable_text
Docker is a powerful tool, however learning how to use it the right way could take a long time especially with the rapidly growing ecosystem of containers which could be confusing, that is why I had the idea to start writing Painless Docker.

Painless Docker is a complete and detailed guide (for beginners and intermediate levels) to create, deploy, optimize, secure, trace, debug, log, orchestrate & monitor Docker and Docker clusters in order to create a good quality microservices applications.

This article is detailed in this book. You can preorder it here.

The latest Docker Swarm release candidate is solving many problems, that's why I decided to test it and see if I can use it in production. I tested Kubernetes but the fact that it is not a built-in tool is an inconvenient even if it has more features. This Docker release integrates Docker Swarm with the Docker Engine.

Let's see in details how I set up a cluster of 3 nodes using Docker 1.2.

Before Docker 1.2, I tested Docker Swarm, your infrastructure architecture for the Swarm cluster will look like this, if you have tried it too:

I am using Digital Ocean for this tutorial but here is what things will look like when using the Swarm Mode with AWS:

3 nodes, each in a different availability zone/subnet and each node is a Swarm manger with one Leader (or primary node) and 2 secondary managers.

Let's see how to implement this using 3 Digital Ocean Droplets.

Creating Swarm Nodes

We start by creating 3 virtual machines in the same private network (if you are using DigitalOcean make sure to activate the "private networking").

You can see that theses machines have 3 public IP addresses but they also have 3 private networking IP addresses and this is what we are going to use:

Node1: 10.136.9.230

Node2: 10.136.12.9

Node3: 10.136.12.10

Private IP addresses will be used for private networking inside the Swarm cluster.

The public IP addresses of our 3 VMs are:

Node1: 192.241.155.28

Node2: 192.241.148.163

Node3: 192.81.216.250

We are going to use this with our sample application.

Installing Docker 1.12, Swarm & Docker Compose

sudo apt-get -y update

apt-get -y install apt-transport-https ca-certificates

apt-key adv - keyserver hkp://p80.pool.sks-keyservers.net:80 - recv-keys 58118E89F3A912897C070ADBF76221572C52609D

echo "debhttps://apt.dockerproject.org/repo ubuntu-trusty experimental" > /etc/apt/sources.list.d/docker.list

sudo apt-get -y update

sudo apt-get purge lxc-docker

sudo apt-get install -y docker-engine

Docker Compose

curl -Lhttps://github.com/docker/compose/releases/download/1.6.2/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose

chmod +x /usr/local/bin/docker-compose

We will use Docker Compose later.

The Swarm Cluster

In this Swarm cluster we made the choice that Node1 will be the leader and that the Swarm Cluster leader will listen on the private network:

docker swarm init --listen-addr 10.136.9.230

Typing this command will output a second command that you should type on the other nodes in order to join the leader:

docker swarm join --secret avekzwl42a9j8vcn8ct0bvsie --ca-hash sha256:81159d56e9ad1cd070dacdec86ee14f9c0b0fb183cac9d5de8a8ab12109aa1e 10.136.9.230:2377

The choice of using private networking is important for your security especially that Docker Swarm is in its beta version.

Anyway, type the last generated command on the other nodes (node2 & node3).

Let's type the generated command on the node2 and see what happens on the node1:

docker swarm join --secret avekzwl42a9j8vcn8ct0bvsie --ca-hash sha256:81159d56e9ad1cd070dacdec86ee14f9c0b0fb183cac9d5de8a8ab12109aa1e 10.136.9.230:2377

This node joined a Swarm as a worker.

Well it says that "This node joined a Swarm as a worker.". Let's go back to the master and verify it:

docker node ls
ID HOSTNAME MEMBERSHIP STATUS AVAILABILITY MANAGER STATUS
9omeucf06tc16u3ifh817jxh2 node2 Accepted Ready Active
at14oahgfhct04mixjod9w2w1 * node1 Accepted Ready Active Leader

Using Docker node command we can list all the nodes that have already joined the cluster.

Do the same thing in node3 and verify again on the master (node1)

docker node ls
ID HOSTNAME MEMBERSHIP STATUS AVAILABILITY MANAGER STATUS
9omeucf06tc16u3ifh817jxh2 node2 Accepted Ready Active
at14oahgfhct04mixjod9w2w1 * node1 Accepted Ready Active Leader
b7c9h10h1rrzgpotqocoykj6j node3 Accepted Ready Active

In a production environment I would think about more security.

Apart from using private networking, you can use some other options when starting a cluster like refusing any auto joining request until approved:

--auto-accept none

Using a validity period for your node certificates could be another security enhancement:

--cert-expiry

You can also give the cluster an explicit secret string:

 --secret string

You can use the Inspect command to get more information about your Swarm cluster:

# docker swarm inspect

[
 {
 "ID": "242c5wqi9xcuihozygj6s9wma",
 "Version": {
 "Index": 662
 },
 "CreatedAt": "2016-07-17T16:49:04.552266747Z",
 "UpdatedAt": "2016-07-17T17:04:41.0267578Z",
 "Spec": {
 "Name": "default",
 "AcceptancePolicy": {
 "Policies": [
 {
 "Role": "worker",
 "Autoaccept": true,
 "Secret": "$2a$10$PKqKQH.fBMeQgygtxqN1F.e/ySMaOn9w1BRHUz40Lt50IHSVoYt3e"
 },
 {
 "Role": "manager",
 "Autoaccept": false,
 "Secret": "$2a$10$PKqKQH.fBMeQgygtxqN1F.e/ySMaOn9w1BRHUz40Lt50IHSVoYt3e"
 }
 ]
 },
 "Orchestration": {
 "TaskHistoryRetentionLimit": 10
 },
 "Raft": {
 "SnapshotInterval": 10000,
 "LogEntriesForSlowFollowers": 500,
 "HeartbeatTick": 1,
 "ElectionTick": 3
 },
 "Dispatcher": {
 "HeartbeatPeriod": 5000000000
 },
 "CAConfig": {
 "NodeCertExpiry": 7776000000000000
 }
 }
 }
]

Using Docker Compose For Development

When I started working with Docker, I run Docker Compose in production to create and scale services. It was, at least for me, quite stable.

I realised also that Docker Compose is the right tool for me to adopt Docker from the development phase to the production.

Developers can use the same container image that runs in production, they get it from the Docker Hub or a private Docker registry. While any modification on the image is pushed to the registry, they can still use exactly the same production configuration and environment.

In this example I am using a simple web application developed by Docker for demonstration purposes.

This is just a sample but it is a reproducible template in order to create a node of complex microservices and a cluster of swarm production nodes.

mkdir -p /apps/app1/

cd /apps/app1

We are going to use the following code for our Docker Compose file:

version: '2'
services:
 webapp:
 container_name: vote
 image: instavote/vote
 ports:
 - 80:80

Let's push this to the Docker Hub. (you can use your own private Docker registry instead).

docker commit -m "first commit" -a "Aymen El Amri" ea534014a41f eon01/vote:v1

docker push eon01/vote:v1

One of the best things about Docker is the fact that images are distributables. In this example I used a web app for demonstration, but let's suppose that we have made some modifications (after mounting the application directory and getting out the code from the container).

Swarm To Deploy Services

Now that we have our Docker Swarm Cluster (Node1 + Node2 + Node3), that we made our modification on the vote app container & code and that everything is ready for deployment, let's create the service

root@node1:~# docker service create --name vote -p 8080:80 eon01/vote:v1

21nd38tu0kpuy52g7cdu4oz23

Let's see what are the services that we run on our Leader machine:

root@node1:~# docker service ls
ID NAME REPLICAS IMAGE COMMAND
1asx5ueen9ii vote 10/10 eon01/vote:v1

Swarm To Scale Services

A single command to scale the existing service that we called "vote".

root@node1:~# docker service scale vote=10

We run the command on the Leader node (Node1). Notice that on the same node when running

docker ps

You will not find all of the ten containers in the same machine, but scaled containers will be deployed to the whole cluster = 3 nodes.

Node1:

root@node1:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1e23f5d89997 eon01/vote:v1 "gunicorn app:app -b " 8 minutes ago Up 8 minutes 80/tcp vote.51.95jujipdhd1dm4trkn8o0kgj3
87783db001fc eon01/vote:v1 "gunicorn app:app -b " 8 minutes ago Up 8 minutes 80/tcp vote.44.aokel9gezo8p9ne8wp52ghm3c
c6e265649145 eon01/vote:v1 "gunicorn app:app -b " 8 minutes ago Up 8 minutes 80/tcp vote.55.efvkiwobwe9w1h7h7ddfsoaor
37f8418cf488 eon01/vote:v1 "gunicorn app:app -b " 8 minutes ago Up 8 minutes 80/tcp vote.12.90m26t7x4jjduwoh41nclyxi4

Node2:

root@node2:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0e4a2a7968ab eon01/vote:v1 "gunicorn app:app -b " 7 minutes ago Up 7 minutes 80/tcp vote.38.adln4aqu0zbrm54kakxyfiejz
f063cf489756 eon01/vote:v1 "gunicorn app:app -b " 7 minutes ago Up 7 minutes 80/tcp vote.85.0rqxmi2h8vnn63rcu94xtv9d1

Node3:

root@node3:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
eb3c50833abc eon01/vote:v1 "gunicorn app:app -b " 7 minutes ago Up 7 minutes 80/tcp vote.69.4509f465ozde6t6yi8ghsa5k1
6baab4ba33d2 eon01/vote:v1 "gunicorn app:app -b " 8 minutes ago Up 8 minutes 80/tcp vote.88.bukyzg40dcgexyditlkyf58p7

You can also use Tasks command from the Leader/Master node (Node1) to list all the nodes and all the containers.

root@node1:~# docker service tasks vote
ID NAME SERVICE IMAGE LAST STATE DESIRED STATE NODE
90m26t7x4jjduwoh41nclyxi4 vote.12 vote eon01/vote:v1 Running 11 minutes ago Running node1
70s3edthlmat8avlnftqn6g9l vote.15 vote eon01/vote:v1 Accepted 3 seconds ago Accepted node2
64devmstxti4rxjdlu3o6tkj1 vote.24 vote eon01/vote:v1 Accepted 4 seconds ago Accepted node3
adln4aqu0zbrm54kakxyfiejz vote.38 vote eon01/vote:v1 Running 10 minutes ago Running node2
aokel9gezo8p9ne8wp52ghm3c vote.44 vote eon01/vote:v1 Running 11 minutes ago Running node1
95jujipdhd1dm4trkn8o0kgj3 vote.51 vote eon01/vote:v1 Running 11 minutes ago Running node1
efvkiwobwe9w1h7h7ddfsoaor vote.55 vote eon01/vote:v1 Running 11 minutes ago Running node1
4509f465ozde6t6yi8ghsa5k1 vote.69 vote eon01/vote:v1 Running 9 minutes ago Running node3
0rqxmi2h8vnn63rcu94xtv9d1 vote.85 vote eon01/vote:v1 Running 10 minutes ago Running node2
bukyzg40dcgexyditlkyf58p7 vote.88 vote eon01/vote:v1 Running 10 minutes ago Running node3

This is the AB test and the difference before and after scaling the services to 5 (note that I haven't done any other optimisations):

Swarm To Inspect Services

root@node1:~# docker service inspect vote

Swarm To Update Running Services

We have already deployed the v1 if you noticed that our repository tag was:

eon01/vote:v1

Now we would like to update the running service with the v2.

docker commit -m "first commit" -a "Aymen El Amri" vote eon01/vote:v2

docker push eon01/vote:v2

We should in this case log to the master and re-deploy while updating the image tag:

root@node1:~# docker service update - image eon01/vote:v2 vote
vote

The last command will update the container from v1 to v2 in the Leader node and on the other slaves.

root@node2:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a2a5d27c9b7a eon01/vote:v2 "gunicorn app:app -b " 27 seconds ago Up 23 seconds 80/tcp vote.88.75v272k1ejipsuhrx849tkhgp
6c17999781ac eon01/vote:v2 "gunicorn app:app -b " 27 seconds ago Up 23 seconds 80/tcp vote.12.ax0nh98as1xmu0q69vcsfq8sq
9549f47cb055 eon01/vote:v2 "gunicorn app:app -b " 27 seconds ago Up 25 seconds 80/tcp vote.85.1wm3ea9rfi6c8v8qlp7h2i4kv
8ce09c29b4dc eon01/vote:v2 "gunicorn app:app -b " 27 seconds ago Up 23 seconds 80/tcp vote.51.etdcuijr8vsx7eyy0ptvlbbzm

Deleting Services

In order to delete a service use the rm command:

docker rm vote

Removing Nodes From Our Cluster

Like a node can join a cluster, it can also leave it.

Let's force the Node2 to leave the cluster:

root@node2:~# docker swarm leave
Node left the swarm.
What about the 10 containers running simultaneously in the 3 nodes ?
root@node1:~# docker  service tasks vote
ID NAME SERVICE IMAGE LAST STATE DESIRED STATE NODE
evhx77i6b1575sj5oxu5b5yer vote.12 vote eon01/vote:v2 Accepted 1 seconds ago Accepted node3
03kxbhenyroxtpqdkom2mn4z6 vote.15 vote eon01/vote:v2 Running 35 minutes ago Running node3
438eruwgle9cwhwav26vrgkhk vote.24 vote eon01/vote:v2 Running 35 minutes ago Running node3
93apj9vyt7hinarvfp9geg7sv vote.38 vote eon01/vote:v2 Running 35 minutes ago Running node3
1xt7e9xm3d6yuuu87o2bpb4zp vote.44 vote eon01/vote:v2 Running 35 minutes ago Running node1
1hg5b5bhnqlpubmldnknz7qxp vote.51 vote eon01/vote:v2 Running about a minute ago Running node1
0vwormex59myle3n918s11ich vote.55 vote eon01/vote:v2 Running 35 minutes ago Running node3
2qm4v0c47qfye2izglixb0h21 vote.69 vote eon01/vote:v2 Running 35 minutes ago Running node1
10dn58my2tbq6kxa534cchcu1 vote.85 vote eon01/vote:v2 Running about a minute ago Running node1
39m8ia95h0d3gtkco6rf1ii2q vote.88 vote eon01/vote:v2 Running about a minute ago Running node1

Notice that the 10 containers are re-distributed again on the 2 left nodes (Node1 & Node3).

What I noticed also that redistribution was not instantaneous which is not a good point for production.

Other Docker Swarm Features

Docker Swarm in its latest version is not just a separated orchestration tool, but a built-in cluster management tool with secure and distributed design. There is no need, for example, to use Nginx, Haproxy or Traefik as a backend load balancer for your microservices.

As you can also notice the RC version has several features:

Cluster management integrated with Docker Engine

Decentralized design

Declarative service model

Scaling

Desired state reconciliation

Multi-host networking

Service discovery

Load balancing

Secure by default

Rolling updates

What is missing ?

The built-in Docker Swarm was just released and there are may be more modifications or enhancements but what I have noticed is that Docker Compose is not yet integrated.

When a node leaves a cluster, it should at least wait for the other nodes to handle the containers running on the disappearing node but I have noticed that typing on the Node2

docker swarm leave

will result in the disappearance of the node from the cluster but few seconds later, I can still see the Node2 as up (So I imagine it is still getting traffic).

root@node1:~# docker  service tasks vote
ID NAME SERVICE IMAGE LAST STATE DESIRED STATE NODE
ax0nh98as1xmu0q69vcsfq8sq vote.12 vote eon01/vote:v2 Running 33 minutes ago Running node2
03kxbhenyroxtpqdkom2mn4z6 vote.15 vote eon01/vote:v2 Running 33 minutes ago Running node3
438eruwgle9cwhwav26vrgkhk vote.24 vote eon01/vote:v2 Running 33 minutes ago Running node3
93apj9vyt7hinarvfp9geg7sv vote.38 vote eon01/vote:v2 Running 33 minutes ago Running node3
1xt7e9xm3d6yuuu87o2bpb4zp vote.44 vote eon01/vote:v2 Running 33 minutes ago Running node1
etdcuijr8vsx7eyy0ptvlbbzm vote.51 vote eon01/vote:v2 Running 33 minutes ago Running node2
0vwormex59myle3n918s11ich vote.55 vote eon01/vote:v2 Running 33 minutes ago Running node3
2qm4v0c47qfye2izglixb0h21 vote.69 vote eon01/vote:v2 Running 33 minutes ago Running node1
1wm3ea9rfi6c8v8qlp7h2i4kv vote.85 vote eon01/vote:v2 Running 33 minutes ago Running node2
75v272k1ejipsuhrx849tkhgp vote.88 vote eon01/vote:v2 Running 33 minutes ago Running node2

What We Have Seen In This Tutorial

In this tutorial we used a single service with three nodes (1 master and two slaves), in Swarm mode, you can easily create containers from this service and scale it to many containers instances.

We have seen also how to use Docker Compose to share images with developers (images are pulled from Docker registry to stay up to date).

We used a single micro service but this is a template that could be easily reproduced with many other micro services.

Actually, I may use this version of Docker Swarm in production but not in critical and high load applications like databases and I am looking forward to discover the final version.

Connect Deeper

If you resonated with this article, please subscribe to DevOpsLinks : An Online Community Of Diverse & Passionate DevOps, SysAdmins & Developers From All Over The World.

You can find me on Twitter, Clarity or my blog and you can also check my books: SaltStack For DevOps, The Jumpstart Up & Painless Docker.

Don't forget to recommend this article to your followers and share it.

# data[].attributes.attachment_link.title
MicroServices From Development To Production Using Docker, Docker Compose & Docker Swarm

# data[].attributes.attachment_link.description
Docker is a powerful tool, however learning how to use it the right way could take a long time especially with the rapidly growing ecosystem of containers which could be confusing, that is why I had the idea to start writing Painless Docker.

# data[].attributes.attachment_link.readable_text
To gather insights on the evolution of IoT to this point in 2017, we spoke to 19 executives who are familiar with the current state of the Internet of Things. Here's who we talked to:

We asked them, "What skills do developers need to be successful working on IoT projects?" Here's what they told us:

Learn the impact of data, how databases work, as well as the data tier. Think about how data applies to the real world. How will data react in a distributed world? One side exists in which technical developers like SQL and Spark but as these come together need to understand streaming data, machine learning, and creating feedback loops to provide more value so the edge is continually getting better.

SQL, Java, Python, distributed systems, machine learning, artificial intelligence. Understand how to collect data and learn from it. Have the ability to ask the right questions to get more out of the data.

The software resides on the device that is resource constricted. Learn to deal with intermittent connected devices. How do you plan for that and manage transitions from one device to another?

Two answers depending on what you're trying to accomplish. 1) If you're a student, understand Amazon's and Azure's approach to IoT and learn their products and tools. 2) To be successful, understand the big picture of all moving parts, not just the bits and bytes of code. We moved from Waterfall engineering out to Agile processes where we are learning and iterating more quickly. Work on teams focused on services that can stand alone and collect from other services. A software development architect will understand the big picture.

Know coding. Listen and execute flawlessly. The key person is the product manager who can work with engineering, developers, and the business professionals. Understand how to solve business problems and add value.

1) Systems programming is back. Edge apps run on different environments. 2) Interconnectivity to get value and end user experience with APIs and DevOps.

From a storage POV, we made it easier to have developer interfaces to storage via a standard Restful. Easy to have Restful-based protocol access.

Think end-to-end solutions, not just embedded solutions. Connect to the cloud and be able to scale. Bringing together security, IoT, and data is challenging. Individuals and teams need to be mindful of the challenge of managing and visualizing data.

Understand what's possible with software, hardware, and gateways.

Take a standards approach to leveraging a broad ecosystem gives developers a lot of flexibility. Understand the use of machine learning algorithms and how they run on physical devices to make decisions in real time. This requires an understanding of microservices.

A lot of platforms are out there that require low level programming experience. Jump in. The need for computer science experience is shrinking. Go find the platforms and start playing with them. Don't start from scratch.

1) Keep minds open because so many different standards around security, scalability and resource management. Code must be efficient in memory, power, and bandwidth consumption. Pick up an Arduino or RaspberryPi and see a light blinking because of the code you wrote. Contribute code to the Open Source community.

Have database management skills and be able to obtain insights from data. The ability to extract patterns from information. Software developers tend to write bloated code. Need to know how to write for resource constrained products. There will be a demand for embedded programmers.

Device developers and manufacturers need to be looking for partners, both in delivery (cable and ISPs) and technology (connectivity, cloud, analytics), that can help them design and develop IoT solutions quickly and cost-effectively. When choosing a connectivity hardware supplier, system developers should select a partner with a proven hardware reference design and a mature, well-documented software development kit (SDK) that will work for their solution and their chosen service provider. For example, if their service provider plans on using a variety of different connectivity standards, then multi-channel radio support in the silicon is critical. Multi-stack protocol capability is key, as is a scalable software architecture that enables the engineer to plug-in different application protocols as needed.

Think about the IoT partners you are working with. Know Java and .Net to be able to hard code the technical resources needed to achieve the business outcomes. Be cognizant of the need to collaborate within your organization, with resource providers, and with clients. Be a full-stack developer that knows how to connect APIs. Stay abreast of the changes in the industry and flexible with regards to platforms, architectures, and languages.

How to work with data, how to architect it, how to aggregate data, model it, make it analytic ready. Access to big data technologies such as MongoDB, Hadoop. Knowledge of data mining and predictive modeling

What else do developers need to keep in mind when working on IoT projects?

And in case you're wondering, here's who we talked to:

Scott Hanson, Founder, and CTO, Ambiq Micro

Adam Wray, CEO and Peter Coppola, SVP, Product Marketing, Basho

Farnaz Erfan, Senior Director, Product Marketing, Birst

Shahin Pirooz, CTO, Data Endure

Anders Wallgren, CTO, Electric Cloud

Eric Free, S.V.P. Strategic Growth, Flexera

Brad Bush, Partner, Fortium Partners

Marisa Sires Wang, Vice President of Product, Gigya

Tony Paine, Kepware Platform President at PTC, Kepware

Eric Mizell, Vice President Global Engineering, Kinetica

Crystal Valentine, PhD, V.P. Technology Strategy and Jack Norris, S.V.P., Database Strategy and Applications, MapR

Pratibha Salwan, S.V.P. Digital Services Americas, NIIT Technologies

Guy Yehaiv, CEO, Profitect

Cees Links, general manager Wireless Connectivity, Qorvo

Paul Turner, CMO, Scality

Harsh Upreti, Product Marketing Manager, API, SmartBear

Rajeev Kozhikkuttuthodi, Vice President of Product Management, TIBCO

# data[].attributes.attachment_link.description
To gather insights on the evolution of IoT to this point in 2017, we spoke to 19 executives who are familiar with the current state of the Internet of Things. Here's who we talked to: We asked them, "What skills do developers need to be successful working on IoT projects?"

# data[].attributes.attachment_link.readable_text
The Internet of Things (IoT) industry is booming-in 2017, the number of connected devices in use worldwide will reach 8.4 billion, outnumbering people, according to a recent Gartner report. By 2020, more than 20.8 billion IoT devices will be in use, Gartner predicts.

As connected homes, cars, and offices become more mainstream, more developers are needed to ensure that devices operate properly and securely.

The term "IoT developer" remains broad, said Greg Gorman, director of the IoT Developer Ecosystem at IBM. "There are a lot of discipline areas that are in play, including security, networking, systems engineering, cloud programming, and hardware device programming," Gorman said. "It pays to be multilingual so that you can be flexible and play many different roles in the team."

There are four stages in developing an IoT device, according to Kornilios Ampatzis, a software developer at InfoLearn:

Assembly of the physical hardware: This requires engineering skills, and is usually not completed by a developer. Most IoT devices use primarily pre-assembled boards and sensors connected on them.

Programming the device: This requires programming skills to read the data from the sensors connected on the IoT device, and send them to the server.

Programming the server that will receive and store the data from the device: This requires the use of server side languages, like PHP, ASP.NET or Node.js, and database queries based on MySQL or some other SQL derivative.

Displaying data to the device user: This involves creating the web page or app that will depict the collected data to the user, which requires web development knowledge of PHP, JavaScript, HTML, CSS, MySQL, or another framework.

"Usually a developer is not responsible for all those stages," Ampatzis said. "So, in order to specify on how to get started on a career in the field, first they have to decide on which stage of the development process they want to get aboard."
SEE: The five industries leading the IoT revolution (ZDNet)

mangpor_2004, Getty Images/iStockphoto

Here are six tips from IoT experts on how to break into a career developing connected devices.

1. Gain a deep understanding of sensors

Unlike other developers, those who work in the IoT space must have a deep understanding of sensors and wireless communication, according to Karen Panetta, an IEEE fellow, and a professor of electrical and computer engineering and associate dean for graduate education at Tufts University.

It's recommended that IoT developers have a background in computer science or electrical engineering, Panetta said. However, IEEE and other professional organizations offer online courses on sensors and development in which you can make a project to show employers. And a number of inexpensive sensors and maker kits are available to practice skills on your own.

"Beyond computing, IoT will take you into the world of mechanical and civil engineering as sensors gather physics data," said Bryan Kester, head of IoT at Autodesk. "It's very difficult to be a 'deep' IoT technologist-you have to be naturally curious about the world and a renaissance person at heart."

2. Focus on user interface

When developing a commercial IoT product, it's important to hold yourself to high quality standards for user experiences, said Kit Klein, head of engineering at Wink. "Many customers depend on these products for critical tasks in their daily lives and are understandably intolerant of failures," Klein said. "As an industry, we need to ensure products delight a rapidly growing base of users who aren't necessarily tech savvy. Quality and reliability are paramount to this experience and need to be part of any developer's mentality."

Panetta recommends performing usability studies with customers to determine ease of use. "It all comes back to user interface," Panetta said. "You can have the best control for your thermostat, but it needs to be simple to use."

3. Learn JavaScript or Python

Suz Hinton, technical evangelist at Microsoft, recommends learning JavaScript before pursuing an IoT developer career. "Using a web based language for both the data processing backend, and the code running on the device itself makes a lot of sense," Hinton said. "JavaScript is a very event-driven language, and this makes it ideal for reacting to new data from devices and triggering actions on the devices themselves."

Working with new technology often means being motivated to work without documentation, code samples, or guidance other than the scant information provided by hardware manufacturers, said Rob Lauer, senior manager of developer relations at Progress. IoT developers tend to use common languages, including Python and JavaScript, with some Windows IoT-compatible devices using C#/.NET, he said.
SEE: How to become a developer: 7 tips from the pros
4. Play with a Raspberry Pi

For those without a computer science or electrical engineering degree, Elliot Schrock, founder and lead developer at Thryv, Inc., suggests demonstrating your aptitude to employers by completing projects on a Raspberry Pi.

"Raspberry Pis are very inexpensive, tiny computers, and are often employed in proof of concept IoT projects," Schrock said. "They're also a great way to learn how to solder together simple circuits, and link those circuits with software. Putting together some simple demo projects and then coming up with, and executing, some projects of your own is a great way to show that you have the initiative and know-how to work in IoT."

Hinton agreed. "Using a device like the Tessel 2, or the Particle Photon, or even the humble Raspberry Pi can get developers fast on their way to learning how hardware ticks and the new skills required," Hinton said. "Writing for IoT is really just learning how to write for smaller, slower computers."

5. Find a community

Involvement in the surrounding communities of makers, inventors, and entrepreneurs with whom one can explore, develop, and refine their ideas into a reality is an important factor for becoming an IoT developer, said Emily Rose, lead developer evangelist at Salesforce. "The world of IoT is still so nascent and nebulous; there are few well-defined paths into the industry," she added. "This may seem like a daunting prospect, but it can also be a tremendous advantage to those with an eye for exploration beyond the bounds of convention."

David Middlecamp, Lead Solution Architect at Particle, suggests looking to Hackster and Instructables for communities of makers and project ideas.

6. Keep your skills cutting edge

Learning one platform or skillset isn't enough, according to IBM research scientist and master inventor Eli Dow. "The platform you write for this week will often be obsolete within 6 months to a year," Dow said. "Sensors will change, single board computers or other embedded platforms will continue to evolve, and you have to have the flexibility to adapt as platforms change at a blistering pace."

Becoming an IoT developer means being "obsessed" with technology, said Erin Essex, creative director at Webonise. "Successful IoT developers must be tech news junkies-they should know everything that is going on in the industry, what's hot, what's old news, and what could be the next great thing," Essex said. "This will provide the foundation needed to tinker with technology and make whatever is being built, the best it can possibly be."

Also see

# data[].attributes.attachment_link.description
The Internet of Things (IoT) industry is booming-in 2017, the number of connected devices in use worldwide will reach 8.4 billion, outnumbering people, according to a recent Gartner report. By 2020, more than 20.8 billion IoT devices will be in use, Gartner predicts.

# data[].attributes.attachment_link.readable_text
For the vast quantity of articles, guides and opinions written about the Internet of Things (IoT), it seems much attention goes to shiny, new "Things," while the humble, foundational "Internet," waits in the wing for its moment in the spotlight. The truth of the matter is that IoT as we know it today takes an ensemble cast and crew to make the show sing. Behind the scenes, developers are hard at work making any number of decisions on how to connect devices to the cloud, how to determine which protocols can be used, how to secure connections and the best methods of authentication.

This post is for the cast and crew of developers that make any IoT solution work for its intended audience. It's a guide not just to the "Things," but to their secure connection to the "Internet" that powers every IoT solution.

What is Azure IoT?

Microsoft's end-to-end IoT platform is the most complete IoT offering, enabling enterprises to build and realize value from IoT solutions quickly and efficiently. Our IoT solutions, including Azure IoT Suite and Azure IoT Hub, harness the power of our full cloud, data and developer offerings for the enterprise to provide hyperscale IoT services, rich data and analytics, and deep integration. Developers can start building custom solutions using IoT Hub or get started quickly with the comprehensive preconfigured solutions of Azure IoT Suite (which includes Azure IoT Hub). This post will focus mostly on Azure IoT Hub.

To help developers leverage the power of IoT in their endeavors and get started quickly, Azure IoT offers a set of open source SDKs, samples, preconfigured solutions and tools. Microsoft engineers work on these SDKs, tools and samples on GitHub, collaborating with the community and partners in the open to add features; support new languages, protocols, and devices and platforms; fix bugs; and improve performance.

The Azure IoT developer center is a great place to start your journey with Azure IoT, and this article will give you a big picture of what to expect when you jump in.

Develop for Azure IoT Hub

Azure IoT Hub provides an easy and secure way to connect, provision and manage millions of IoT devices sending and receiving billions of messages per month. IoT Hub is the bridge between your devices and their solutions in the cloud, allowing them to store, analyze and act on that data in real time. IoT Hub enables secure, reliable, two-way communication - from device to cloud and cloud to device - over open protocols such as MQTT, HTTPS and AMQPS that are already widely used in IoT.

As a developer you will be looking at three main objectives: connecting devices to IoT Hub, managing the IoT Hub service itself and integrating IoT Hub into your overall IoT solution in the cloud.

To achieve all three, the Azure IoT Hub service exposes REST APIs along with AMQP and MQTT communication support, but implementing communication protocols is not trivial by nature. We've got you covered on both the service and the device client sides with open source SDKs and samples that abstract the complexity of these protocols and expose simple and straightforward APIs. All our client SDKs are open sourced on GitHub and offer the following:

Device client:

Support for AMQP, AMQP over WebSockets, MQTT and HTTP/REST for the device-to-cloud communication

Support for SSL (using third-party dependencies such as openSSL or WolfSSL)

Simple APIs to:

Establish a secure connection to IoT Hub

Send messages to IoT Hub

Receive messages from IoT Hub

Service client:
Simple APIs to:

Manage the device registry (Create, Remove, Update, Delete)

Read data from IoT Hub

Send messages to specific devices

For connecting devices that cannot talk to the protocols supported by IoT Hub, there is an open source sample for a cloud protocol gateway that serves as a bridge between the protocol of your choice and IoT Hub.

The diagram below shows where the open source SDKs and samples can be used.

Connecting securely to Azure IoT Hub

Azure IoT Hub offers a secure mechanism for connecting devices.

First is the notion of access control. The Azure IoT Hub service uses a set of permissions to grant access to endpoints. Developers can set up permissions using shared access policies for services, apps and devices and can create per-device security credentials leveraging the device identity registry feature of the service.

Then Azure IoT Hub will authenticate endpoints by verifying a token against the shared access policies and device identity registry security credentials. Security credentials, such as symmetric keys, are never sent over the wire. The security token generation is implemented in the device client SDKs.

Last but not least, all endpoints connect to Azure IoT Hub over TLS, ensuring no endpoint is ever exposed on unencrypted or unsecured channels.

For devices, the device client SDKs implement all of this under the hoods for you, so as a developer you can focus on the development of the actual solution trusting that the plumbing is robust and secured.

Connecting a device is as simple as the below line of C code:
iotHubClientHandle = IoTHubClient_CreateFromConnectionString(connectionString, AMQP_Protocol);
For managing the service from an application, the service client SDKs also implement all that's needed to connect securely to Azure IoT Hub.

To learn more on communication security for Azure IoT Hub, read the security section of the service developer's guide.

Sending and receiving messages

Once a secure connection is established from devices to Azure IoT Hub, all that's left to do is to send and receive messages. Azure IoT Hub offers a raw messaging infrastructure, meaning you can put whatever you want and need in the body of your messages.

Sending messages is as simple as the C code snippet below:
msgHandle = IoTHubMessage_CreateFromByteArray(msgText, strlen(msgText)); IoTHubClient_SendEventAsync(iotHubClientHandle, msgHandle, SendConfirmationCallback, NULL);
And if you want to receive messages from Azure IoT Hub on the device, you need to register a callback like in the below C example:
IoTHubClient_SetMessageCallback(iotHubClientHandle, ReceiveMessageCallback, NULL);
The SDKs also allow to know and notify when a message has been received on the other side.

You will find more details in the various samples in the SDKs repository for each of the languages ( C, JavaScript (node), Python, Java and C#).

Packages, tools, documentation and samples to make your life easier

In addition to simple APIs, developers will benefit from binary packages, tools, documentation and samples, all part of the SDKs open source repository, to get started faster and implement and debug their solutions efficiently.

Binary packages such as npm for JavaScript, NuGet for C#, apt-get for some Linux Distributions, Maven for Java and pip wheels for Python saves you from having to build and compile the whole SDK project on your target device if not needed. Running a sample on a device using one of these packages is just a few easy steps.

In the process of creating and debugging your IoT solution with Azure IoT Hub, you will need to configure access credentials in the IoT Hub service, create device identities, monitor messages coming from devices and send messages to devices to test your code. To achieve these tasks easily, convinient tools are provided:

The Azure CLI IoT extension adds functionality specific to IoT Hub to the Azure to the Python based Azure CLI 2.0.

Visual Studio Code has a great extension to help you with your IoT Hub development offering

Device Explorer is a Windows application that comes with a user interface.

When it comes to documentation, you can find the following:

Last but not least, tons of open source samples are available in the SDKs repositories for you to get started faster. Simple samples show, for each language, how each of the supported protocols work to send and receive messages, while more advanced ones show how to connect to Azure IoT Suite or how to implement a simple temperature anomaly detection system.

End-to-end samples are also published on the Azure samples portal by the broader community that you can get inspired by as well.

Hardware and software requirements

You can connect a wide range of devices directly to Azure IoT Hub. Your device will have to have enough resources to have an IP address and establish a TCP over SSL connection. The device client SDKs support various platforms depending on the languages: microcontrollers, real-time OS, Linux, Windows, etc. You can find an exhaustive list of all the supported OS platforms, HW platforms tested, protocols here.

If you prefer shopping for hardware that already supports Azure IoT connectivity, check out the extensive list of supported platforms along with a list of the boards certified for Azure IoT Hub.

When it comes to the OS you will be developing on, you can work on Windows, Linux or OSX, depending on the devices you are targeting and whether or not you intend to compile the SDKs yourself or use one of the binary packages. You can find details on how to set up your development environment in the various sample instructions.

Developing for Azure IoT Suite

IoT Hub is an amazing, foundational service for a customized IoT solution, but it is just the start. Microsoft goes beyond building-block services by providing preconfigured IoT solutions, so that what used to take weeks for a customer to build can now be automatically provisioned in minutes with Azure IoT Suite. Azure IoT Suite enables customers to quickly provision a working, end-to-end IoT solution that includes IoT Hub as well as other Azure services - and even includes simulated devices to help you get started rapidly prototyping your end-to-end solution with a one-click deploy in minutes.

Developing for Azure IoT Suite is divided in two simple steps:

Connecting devices to a solution, which consists of connecting devices to Azure IoT Hub and ensure the devices are sending the data in the right format for the IoT Suite solution instance. For each of the Azure IoT Suite preconfigured solutions, you will find samples and instructions for connecting devices.

Customizing the services deployed as part of the Azure IoT Suite preconfigured solution. All the preconfigured solutions are provided as source code, which allows you to modify, adapt and redeploy every single part of the solution. Extensive documentation can be found here.

And more ...

Azure IoT Hub and Azure IoT Suite are obviously not the only things you will want to work with and develop against when creating an end-to-end solution. There are plenty other services you can utilize, benefit from and spin up on Azure to create your IoT solution. A good starting point for a first implementation of an end-to-end IoT solution is Azure IoT Suite, as you will very easily deploy a set of services that are pre-configured and up and running in a few minutes.

What's next?

This was only the introduction to what Azure IoT has to offer to developers, from securely connecting "Thing" to the "Internet" to sending and receiving messages and debugging your IoT solutions, SDKs, tools, samples and documentation. This will allow you to focus on what counts: your idea for tomorrow's next big IoT solution.

Visit our Azure IoT developer center, check out some of the introduction videos on IoT Hub or IoT Suite, scan the list of certified hardware, try out some of our tutorials and samples, and as always, if you have questions, suggestions or issues, come meet the Azure IoT dev team on GitHub.

# data[].attributes.attachment_link.description
This post is for the cast and crew of developers that make any IoT solution work for its intended audience. It's a guide not just to the "Things," but to their secure connection to the "Internet" that powers every IoT solution.

# data[].attributes.attachment_link.description
Modern software-based services are implemented as distributed systems with complex behavior and failure modes. Many large tech organizations are using experimentation to verify such systems' reliability. Netflix engineers call this approach chaos engineering. They've determined several principles underlying it and have used it to run experiments.

# data[].attributes.attachment_link.readable_text
Last Updated June 12th, 2018

With the rise of microservices and distributed cloud architectures, the web has grown increasingly complex. We all depend on these systems more than ever, yet failures have become much harder to predict.

These failures cause costly outages for companies. The outages hurt customers trying to shop, transact business, and get work done. Even brief outages can impact a company's bottom line, so the cost of downtime is becoming a KPI for many engineering teams. For example, in 2017, 98% of organizations said a single hour of downtime would cost their business over $100,000. One outage can cost a single company millions of dollars. The CEO of British Airways recently explained how one failure that stranded tens of thousands of British Airways (BA) passengers in May 2017 cost the company 80 million pounds ($102.19 million USD).

Companies need a solution to this challenge-waiting for the next costly outage is not an option. To meet the challenge head on, more and more companies are turning to Chaos Engineering.
Chaos Engineering is Preventive Medicine
Chaos Engineering is a disciplined approach to identifying failures before they become outages. By proactively testing how a system responds under stress, you can identify and fix failures before they end up in the news.

Chaos Engineering lets you compare what you think will happen to what actually happens in your systems. You literally "break things on purpose" to learn how to build more resilient systems.

Chaos engineering first became relevant at internet companies that were pioneering large scale, distributed systems. These systems were so complex that they required a new approach to test for failure.

2010

The Netflix Eng Tools team created Chaos Monkey. Chaos Monkey was created in response to Netflix's move from physical infrastructure to cloud infrastructure provided by Amazon Web Services, and the need to be sure that a loss of an Amazon instance wouldn't affect the Netflix streaming experience.

2011

The Simian Army was born. The Simian Army added additional failure injection modes on top of Chaos Monkey that would allow testing of a more complete suite of failure states, and thus build resilience to those as well. "The cloud is all about redundancy and fault-tolerance. Since no single component can guarantee 100% uptime (and even the most expensive hardware eventually fails), we have to design a cloud architecture where individual components can fail without affecting the availability of the entire system" ( Netflix, 2011).

2012

Netflix shared the source code for Chaos Monkey on Github, saying that they "have found that the best defense against major unexpected failures is to fail often. By frequently causing failures, we force our services to be built in a way that is more resilient" ( Netflix, 2012).

2014

Netflix decided they would create a new role: the Chaos Engineer. Bruce Wong coined the term, and Dan Woods shared it with the greater engineering community via Twitter. Dan Woods explained, "I learned more about chaos engineering from Kolton Andrus than anyone else, he called it failure injection testing".

In October of 2014, while Gremlin co-founder Kolton Andrus was at Netflix, his team announced Failure Injection Testing (FIT), a new tool that built on the concepts of the Simian Army, but gave developers more granular control over the "blast radius" of their failure injection. The Simian Army tools had been so effective that in some instances they created painful outages, causing many Netflix developers to grow wary of them. FIT gave developers control over the scope of their failure so they could realize the insights of Chaos Engineering, but mitigate potential downside.

Chaos Engineering involves running thoughtful, planned experiments that teach us how our systems behave in the face of failure.

These experiments follow three steps:

You start by forming a hypothesis about how a system should behave when something goes wrong.

Then, you design the smallest possible experiment to test it in your system.

Finally, you measure the impact of the failure at each step, looking for signs of success or failure. When the experiment is over, you have a better understanding of your system's real-world behavior.

Many larger tech companies practice Chaos Engineering to better understand their distributed systems and microservice architectures. The list includes Twilio, Netflix, LinkedIn, Facebook, Google, Microsoft, Amazon, and many others. The list is always growing.

But more traditional industries, like banking and finance, have caught on to Chaos Engineering, too. For example, in 2014, the National Australia Bank migrated from physical infrastructure to Amazon Web Services and used Chaos Engineering to dramatically reduce incident counts.

The Chaos Engineering Slack Community has created a diagram that tracks known Chaos Engineering tools and known engineers working on Chaos Engineering.

It's helpful to think of a vaccine or a flu shot where you inject yourself with a small amount of a potentially harmful foreign body in order to prevent illness. Chaos Engineering is a tool we use to build such an immunity in our technical systems by injecting harm (like latency, CPU failure, or network black holes) in order to find and mitigate potential weaknesses.

These experiments have the added benefit of helping teams build muscle memory in resolving outages, akin to a fire drill (or changing a flat tire, in the Netflix analogy). By breaking things on purpose we surface unknown issues that could impact our systems and customers.

These posts can help you get started with Chaos Engineering:

Distributed systems are inherently more complex than monolithic systems, so it's hard to predict all the ways they might fail. The eight fallacies of distributed systems shared by Peter Deutsch and others at Sun Microsystems describe false assumptions that programmers new to distributed applications invariably make.

Fallacies of Distributed Systems:

The network is reliable

Latency is zero

Bandwidth is infinite

The network is secure

Topology doesn't change

There is one administrator

Transport cost is zero

The network is homogeneous

Many of these fallacies drive the design of Chaos Engineering experiments such as "packet-loss attacks" and "latency attacks". For example, network outages can cause a range of failures for applications that severely impact customers. Applications may stall while they wait endlessly for a packet. Applications may permanently consume memory or other Linux system resources. And even after a network outage has passed, applications may fail to retry stalled operations, or may retry too aggressively. Applications may even require a manual restart. Each of these examples need to be tested and prepared for.

Customer: the increased availability and durability of service means no outages disrupt their day-to-day lives.

Business: Chaos Engineering can help prevent extremely large losses in revenue and maintenance costs, create happier and more engaged engineers, improve in on-call training for engineering teams, and improve the SEV (incident) Management Program for the entire company.

Technical: the insights from chaos experiments can mean a reduction in incidents, reduction in on-call burden, increased understanding of system failure modes, improved system design, faster mean time to detection for SEVs, and reduction in repeated SEVs.

Many engineering organizations, including Netflix and Stitch Fix, have dedicated Chaos Engineering teams. These teams are often small in size, with 2-5 engineers. The Chaos Engineering team owns and advocates for Chaos Engineering across the organization. However, they are not the only engineers doing Chaos Engineering day-to-day-they empower teams across their engineering organization to use Chaos Engineering.

These service teams are often the first to practice and evangelize Chaos Engineering within a company:

Traffic Team (e.g. Nginx, Apache, DNS)

Streaming Team (e.g. Kafka)

Storage Team (e.g. S3)

Data Team (e.g. Hadoop/HDFS)

Database Team (e.g. MySQL, Amazon RDS, PostgreSQL)

Some companies, such as Remind, are integrating Chaos Engineering into their normal release cycle like other best practice testing as a way to ensure reliability is baked into every feature.

We argue that you should perform your experiments in the following order:

1. Known Knowns - Things you are aware of and understand

2. Known Unknowns - Things you are aware of but don't fully understand

3. Unknown Knowns - Things you understand but are not aware of

4. Unknown Unknowns - Things you are neither aware of nor fully understand

The diagram below illustrates this concept:

To illustrate this in practice with examples, we will demonstrate how to select experiments based on a sharded MySQL Database. In this example, we have a cluster of 100 MySQL hosts with multiple shards per host.

In one region, we have a primary database host with two replicas and we use semi-sync replication. We also have a pseudo primary and two pseudo replicas in a different region.

Known-Knowns

We know that when a replica shuts down it will be removed from the cluster. We know that a new replica will then be cloned from the primary and added back to the cluster.

Known-Unknowns

We know that the clone will occur, as we have logs that confirm if it succeeds or fails, but we don't know the weekly average of the mean time it takes from experiencing a failure to adding a clone back to the cluster effectively.

We know we will get an alert that the cluster has only one replica after 5 minutes but we don't know if our alerting threshold should be adjusted to more effectively prevent incidents.

Unknown-Knowns

If we shutdown the two replicas for a cluster at the same time, we don't know exactly the mean time during a Monday morning it would take us to clone two new replicas off the existing primary. But we do know we have a pseudo primary and two replicas which will also have the transactions.

Unknown-Unknowns

We don't know exactly what would happen if we shutdown an entire cluster in our main region, and we don't know if the pseudo region would be able to failover effectively because we have not yet run this scenario.

We would create the following chaos experiments, working through them in order:

1. Known-Knowns: shut down one replica and measure the time it takes for the shutdown to be detected, the replica to be removed, the clone to kick-off, the clone to be completed, and the clone to be added back to the cluster. Before you kick off this experiment increase replicas from two to three. Run the shutdown experiment at a regular frequency but aim to avoid the experiment resulting in 0 replicas at any time. Report on the mean total time to recovery for a replica shutdown failure and break this down by day and time to account for peak hours.

2. Known-Unknowns: Use the results and data of the known-known experiment to answer questions which would currently be "known-unknowns". You will now be able to know the impact the weekly average of the mean time it takes from experiencing a failure to adding a clone back to the cluster effectively. You will also know if 5 minutes is an appropriate alerting threshold to prevent SEVs.

3. Unknown-Knowns: Increase the number of replicas to four before conducting this experiment. Shutdown two replicas for a cluster at the same time, collect the mean time during a Monday morning over several months to determine how long it would take us to clone two new replicas off the existing primary. This experiment may identify unknown issues, for example, the primary cannot handle the load from cloning and backups at the same time and you need to make better use of the replicas.

4. Unknown-Unknowns: Shut down of an entire cluster (primary and two replicas) would require engineering work to make this possible. This failure may occur unexpectedly in the wild, but you are not yet ready to handle it. Prioritize the engineering work to handle this failure scenario before you perform chaos experiments.

Planning your First Experiment

One of the most powerful questions in Chaos Engineering is "What could go wrong?". By asking this question about our services and environments, we can review potential weaknesses and discuss expected outcomes. Similar to a risk assessment, this informs priorities about which scenarios are more likely (or more frightening) and should be tested first. By sitting down as a team and whiteboarding your service(s), dependencies (both internal and external), and data stores, you can formulate a picture of "What could go wrong?". When in doubt, injecting a failure or a delay into each of your dependencies is a great place to start.

Creating a Hypothesis

You have an idea of what can go wrong. You have chosen the exact failure to inject. What happens next? This is an excellent thought exercise to work through as a team. By discussing the scenario, you can hypothesize on the expected outcome before running it live. What will be the impact to customers, to your service, or to your dependencies?

Measuring the Impact

To understand how your system behaves under stress, you need to measure your system's availability and durability. It is good to have a key performance metric that correlates to customer success (such as orders per minute, or stream starts per second). As a rule of thumb, if you ever see an impact to these metrics, you want to halt the experiment immediately. Next is measuring the failure itself where you want to verify (or disprove) your hypothesis. This could be the impact on latency, requests per second, or system resources. Lastly, you want to survey your dashboards and alarms for unintended side effects.

Have a Rollback Plan

Always have a backup plan in case things go wrong, but accept that sometimes even the backup plan can fail. Talk through how you're going to revert the impact. If you're running commands by hand, be thoughtful not to break ssh or control plane access to your instances. One of the core aspects of Gremlin is safety. All of our attacks can be reverted immediately, allowing you to safely abort and return to steady state if things go wrong.

Go fix it!

After running your first experiment, hopefully, there is one of two outcomes: either you've verified that your system is resilient to the failure you introduced, or you've found a problem you need to fix. Both of these are good outcomes. On one hand, you've increased your confidence in the system and its behavior, and on the other you've found a problem before it caused an outage.

Have Fun

Chaos Engineering is a tool to make your job easier. By proactively testing and validating your system's failure modes you will reduce your operational burden, increase your availability, and sleep better at night. Gremlin makes it safe and simple to get started-email us to get started today!

Pavlos Ratis has created a GitHub repo called " Awesome Chaos Engineering," which is a curated list of Chaos Engineering resources. You can find Books, Tools, Papers, Blogs, Newsletters, Conferences, MeetUps, Forums and engineers to follow on Twitter.

Gremlin Principal Software Engineer Matt Jacobs has written a guide on 4 Chaos Engineering Experiments To Start With.

The Chaos Engineering community is global, with engineers based in over 10 countries around the world.

As web systems have grown much more complex with the rise of distributed systems and microservices, system failures have become difficult to predict. So in order to prevent failures from happening, we all need to be proactive in our efforts to learn from failure.

In this paper, we shared a brief history of Chaos Engineering and demonstrated how Chaos Engineering offers you new insights into your systems.

We look forward to hearing about your Chaos Engineering journey and encourage you to share your progress with the Chaos Engineering community.

# data[].attributes.attachment_link.title
Chaos Engineering: the history, principles, and practice

# data[].attributes.attachment_link.description
Last Updated June 12th, 2018 With the rise of microservices and distributed cloud architectures, the web has grown increasingly complex. We all depend on these systems more than ever, yet failures have become much harder to predict. These failures cause costly outages for companies.

# data[].attributes.attachment_link.description
Modern software-based services are implemented as distributed systems with complex behavior and failure modes. Many large tech organizations are using experimentation to verify such systems' reliability. Netflix engineers call this approach chaos engineering. They've determined several principles underlying it and have used it to run experiments.

# data[].attributes.attachment_link.readable_text
All of the data you need,
in one request

GraphQL is an open spec for a flexible API layer. Put GraphQL over your existing backends to build products faster than ever before.

Start Exploring Official Documentation

Faster frontend development

Iterate quickly on apps without waiting for new backend endpoints. Simplify data fetching and management code by getting the data in the shape you need.

Use your existing data

You can use GraphQL on top of your existing infrastructure: REST, SOAP, existing databases, or anything else. Organize your data into a clean, unified API and query it all at once.

Fewer bytes and roundtrips

Make your apps more responsive than ever before by only loading the data you're actually using, and reduce the number of roundtrips to fetch all of the resources for a particular view.

Using GraphQL is easy

Got a few minutes? Get started with GraphQL today.

GraphQL, REST, or both?

GraphQL can live right alongside your existing architecture investments and bring critical improvements to your workflow.

Tools and libraries for every occasion

The incredibly active GraphQL community has produced servers, clients, and developer tools for almost every situation.

Server implementations

Use these technologies to implement a GraphQL API. Full list here.

Client integrations

Easily consume GraphQL data in a way that feels natural for your client.

Developer tools and services

Write queries, visualize your schema, and integrate with your editor.

The API for modern apps

GraphQL is an API technology that meets the demanding requirements of today's frontend apps.

Want to learn more?

Learn from the experts how to implement GraphQL in production and solve common problems.

# data[].attributes.attachment_link.description
All of the data you need,in one request GraphQL is an open spec for a flexible API layer. Put GraphQL over your existing backends to build products faster than ever before. Start Exploring Official Documentation Iterate quickly on apps without waiting for new backend endpoints.

# data[].attributes.attachment_link.readable_text
No matter how large or small your application is, you'll have to deal with fetching data from a remote server at some point. On the front end, this usually involves hitting a REST endpoint, transforming the response, caching it, and updating your UI. For years, REST has been the status quo for APIs, but over the past year, a new API technology called GraphQL has exploded in popularity due to its excellent developer experience and declarative approach to data fetching.

In this post, we'll walk through a couple of hands-on examples to show you how integrating GraphQL into your application will solve many pain points working with remote data. If you're new to GraphQL, don't panic! I'll also highlight some resources to help you learn GraphQL using the Apollo stack, so you can start off 2018 ahead of the curve.

GraphQL 101

Before we dive into how GraphQL makes your life as a front end developer easier, we should first clarify what it is. When we talk about GraphQL, we're either referring to the language itself or its rich ecosystem of tools. At its core, GraphQL is a typed query language developed by Facebook that allows you to describe your data requirements in a declarative way. The shape of your result matches the shape of your query: in the example below, we can expect to receive back an object with a currency property and a rates property containing an array of objects with both currency and rate keys.

{ rates(currency: "USD") { currency rates { currency rate } } }

When we talk about GraphQL in a broader sense, we're often referring to the ecosystem of tools that help you implement GraphQL in your application. On the backend, you'll use Apollo Server to create a GraphQL server, which is a single endpoint that parses a GraphQL request and returns data. How does the server know which data to return? You'll use GraphQL Tools to build a schema (like a blueprint for your data) and a resolver map (just a series of functions that retrieve your data from a REST endpoint, database, or wherever else you choose).

This all sounds more complicated than it actually is - with Apollo Launchpad, a GraphQL server playground, you can create a working GraphQL server in your browser in less than 60 lines of code! 😮 We'll reference this Launchpad I created that wraps the Coinbase API throughout this post.

You'll connect your GraphQL server to your application with Apollo Client, a fast and flexible client that fetches, caches, and updates your data for you. Since Apollo Client isn't coupled to your view layer, you can use it with React, Angular, Vue, or plain JavaScript. Not only is Apollo cross-framework, it's also cross-platform, with React Native & Ionic supported out of the box.

Let's give it a try! 🚀

Now that you're well-versed in what GraphQL is, let's get our hands dirty with a couple of practical examples that illustrate what it's like to develop your front end with Apollo. By the end, I think you'll be convinced that a GraphQL-based architecture with Apollo can help you ship features faster than before.

1. Add new data requirements without adding a new endpoint

We've all been here before: You spend hours building a perfect UI component when suddenly, product requirements change. You quickly realize that the data you need to fulfill these new requirements would either require a complicated waterfall of API requests or worse, a new REST endpoint. Now blocked on your work, you ask the backend team to build you a new endpoint just to satisfy the data needs for one component.

This common frustration no longer exists with GraphQL because the data you consume on the client is no longer coupled to an endpoint's resource. Instead, you always hit the same endpoint for your GraphQL server. Your server specifies all of the resources it has available via your schema and lets your query determine the shape of the result. Let's illustrate these concepts using our Launchpad from before:

In our schema, look at lines 22-26 where we define our ExchangeRate type. These fields list out all the available resources we can query in our application.

type ExchangeRate { currency: String rate: String name: String }

With REST, you're limited to the data your resource provides. If your /exchange-rates endpoint doesn't include name, then you'll need to either hit a different endpoint like /currency for the data or create it if it doesn't exist.

With GraphQL, we know that name is already available to us by inspecting our schema, so we can query for it in our application. Try running this example in Launchpad by adding the name field on the right side panel!

{ rates(currency: "USD") { currency rates { currency rate name } } }

Now, remove the name field and run the same query. See how the shape of our result changes?

Your GraphQL server always gives you back exactly the data you ask for. Nothing more. This differs significantly from REST, where you often have to filter and transform the data you get back from the server into the shape your UI components need. Not only does this save you time, it also results in smaller network payloads and CPU savings from loading and parsing the response.

2. Reduce your state management boilerplate

Fetching data almost always involves updating your application's state. Typically, you'll write code to track at least three actions: one for when the data is loading, one if the data successfully arrives, and one if the data errors out. Once the data arrives, you have to transform it into the shape your UI components expect, normalize it, cache it, and update your UI. This process can be repetitive, requiring countless lines of boilerplate to execute one request.

Let's see how Apollo Client eliminates this tiresome process altogether by looking at an example React app in CodeSandbox. Navigate to `list.js` and scroll to the bottom.

export default graphql(ExchangeRateQuery, { props: ({ data }) => { if (data.loading) { return { loading: data.loading }; } if (data.error) { return { error: data.error }; } return { loading: false, rates: data.rates.rates }; } })(ExchangeRateList);

In this example, React Apollo, Apollo Client's React integration, is binding our exchange rate query to our ExchangeRateList component. Once Apollo Client executes that query, it tracks loading and error state automatically and adds it to the data prop. When Apollo Client receives the result, it will update the data prop with the result of the query, which will update your UI with the rates it needs to render.

Under the hood, Apollo Client normalizes and caches your data for you. Try clicking some of the currencies in the panel on the right to watch the data refresh. Now, select a currency a second time. Notice how the data appears instantaneously? That's the Apollo cache at work! You get all of this for free just by setting up Apollo Client with no additional configuration. 😍 To see the code where we initialize Apollo Client, check out `index.js`.

3. Debug quickly & painlessly with Apollo DevTools & GraphiQL

It looks like Apollo Client does a lot for you! How do we peek inside to understand what's going on? With features like store inspection and full visibility into your queries & mutations, Apollo DevTools not only answers that question, but also makes debugging painless and, dare I say it, fun! 🎉 It's available as an extension for both Chrome and Firefox, with React Native coming soon.

If you want to follow along, install Apollo DevTools for your preferred browser and navigate to our CodeSandbox from the previous example. You'll need to run the example locally by clicking Download in the top nav bar, unzipping the file, running npm install, and finally npm start. Once you open up your browser's dev tools panel, you should see a tab that says Apollo.

First, let's check out our store inspector. This tab mirrors what's currently in your Apollo Client cache, making it easy to confirm your data is stored on the client properly.

Apollo DevTools also enables you to test your queries & mutations in GraphiQL, an interactive query editor and documentation explorer. In fact, you already used GraphiQL in the first example where we experimented with adding fields to our query. To recap, GraphiQL features auto-complete as you type your query into the editor and automatically generated documentation based on GraphQL's type system. It's extremely useful for exploring your schema, with zero maintenance burden for developers.

Try executing queries with GraphiQL in the right side panel of our Launchpad. To show the documentation explorer, you can hover over fields in the query editor and click on the tooltip. If your query runs successfully in GraphiQL, you can be 100% positive that the same query will run successfully in your application.

Level up your GraphQL skills

If you made it to this point, awesome job! 👏 I hope you enjoyed the exercises and got a taste of what it would be like to work with GraphQL on the front end.

Hungry for more? 🌮 Make it your 2018 New Year's resolution to learn more about GraphQL, as I expect its popularity to grow even more in the upcoming year. Here's an example app to get you started featuring the concepts we learned today:

Go forth and GraphQL (and be sure to tag us on Twitter @apollographql along the way)! 🚀

# data[].attributes.attachment_link.title
A Front End Developer's Guide to GraphQL | CSS-Tricks

# data[].attributes.attachment_link.description
No matter how large or small your application is, you'll have to deal with fetching data from a remote server at some point. On the front end, this

# data[].attributes.attachment_link.readable_text
Let's say you're a developer working for a retailer who's built out a catalog of 250 commerce-related APIs, exposing every granular piece of data (product, category, price, inventory, etc) and functionality (add to cart, calculate discounts, reserve inventory, etc). Say you're re-designing your website and need to build a new product detail page. Today's product detail pages need to access data (products, related products, inventory, pricing, etc) from dozens or even hundreds of APIs.

You could always call directly to each of the APIs but for performance (and many other) reasons you don't want to make hundreds of calls from an end-customer's browser back to your APIs. You could always write an intermediary layer whose job it is to query all of the APIs, but now you've tightly coupled this new layer back to your APIs, which goes against the whole philosophy of API and microservice-based development.

To make matters worse, you practically never want to retrieve an entire object from an API. When you retrieve a product, you probably just want its name, description and a few other pieces of information, though there may be hundreds of other attributes.

This is where GraphQL comes in. Facebook had these exact same challenges, only worse because many of its users are on bandwidth and performance-constrained mobile devices. Imagine the number of API calls to build a user's Facebook timeline, now imagine performing all of those queries from an Apple Watch over a cellular internet connection. Facebook built specification for how to query for data called GraphQL. GraphQL.org, the home of the project, states that GraphQL is "A query language for your APIs." Facebook has been using it internally since 2012 and publicly released it in 2015. Since then, it's quickly caught on and is now used by Twitter, Yelp, GitHub, Intuit, Pinterest and New York Times, among many others.

With GraphQL, you just make a single query, specifying exactly what information you want to retrieve. The GraphQL layer then makes requests to the individual APIs to fulfill the request. As a developer, you get a single response containing everything you need to render your new product detail page. Think of GraphQL almost like SQL, where you can retrieve data from multiple database tables using one query. While GraphQL started with queries, it now supports mutations which allows you to create/update/delete data, again similar to SQL.

The advantages of GraphQL are manifold. As just mentioned, developers can access only the exact data they need, which improves performance and reduces the amount of traffic flowing through your network, which is especially helpful for low-bandwidth environments like smartphones and wearable devices. Another advantage is that developers don't have to know how to call each API.

In a microservices environment where teams have some autonomy in how they implement things like security, it can be difficult for developers to call dozens or even hundreds of different APIs, each with a slight (or major) variation. With GraphQL, developers don't have to even know that the APIs exist - they just query GraphQL and that layer does everything for them. GraphQL also reduces coupling between front-ends and back-ends.

Rather than building a separate API and microservice for rendering the product detail page for an Apple Watch (and every other client), you can just make direct queries from your front-end to your GraphQL layer, bypassing queries to these intermediary layers you'd otherwise have to build. Finally, GraphQL pairs nicely with React, which also came from Facebook and is widely seen as the default for front-end development today.

Remember that GraphQL is a specification, not a product. Think of it like HTML, where the World Wide Web Consortium (W3C) defines the specification and then individual browsers are left to implement the code for rendering web pages. In the case of GraphQL, Facebook publishes the specification and individual developers write code that conforms to the specification.

It's important to note that GraphQL a complement to APIs, not a replacement to APIs. Individual teams still need to build applications and expose functionality over APIs, with GraphQL acting as the layer that developers interact with. GraphQL then interacts with the APIs and deals with security and the oddities of each API.

Like with any technology, there are negatives too. It's another layer to maintain. The specification is rapidly maturing and could change as it finalizes. Facebook has sole control over the specification and there are some unresolved intellectual property issues that could be bad if Facebook decided to be malevolent. It's still relatively early in its adoption cycle.

Even with the few downsides, GraphQL is worth evaluating now. It could very well end up being the differentiator that sets your commerce initiatives apart.
This article is published as part of the IDG Contributor Network. Want to Join?

# data[].attributes.attachment_link.title
GraphQL: The API technology to watch in 2018

# data[].attributes.attachment_link.description
Let's say you're a developer working for a retailer who's built out a catalog of 250 commerce-related APIs, exposing every granular piece of data (product, category, price, inventory, etc) and functionality (add to cart, calculate discounts, reserve inventory, etc). Say you're re-designing your website and need to build a new product detail page.

# data[].attributes.body
Name: Fatimah Alyami
StudentID: 19070633

# data[].attributes.body
Name: Huy Ha
Student ID: 19064314

# data[].attributes.headline
Developing applications using Microservices and Kubernetes

# data[].attributes.subject
Developing applications using Microservices and Kubernetes

# data[].attributes.body
Name: Thantrige Rasika Hemantha
StudentID: 19064839

# data[].attributes.body
Name: Ruby Rana
Student ID: 20094493

# data[].attributes.headline
Developing using Serverless computing (Lambda)

# data[].attributes.subject
Developing using Serverless computing (Lambda)

# data[].attributes.body
Name: Jian (Jason) Chen
Student ID:20109091

# data[].attributes.headline
Security of image recognition in applications

# data[].attributes.subject
Security of image recognition in applications

# data[].attributes.body
Name: Fan Yang
Student ID: 20104813

# data[].attributes.headline
Developing applications using Microservices and Kubernetes

# data[].attributes.subject
Developing applications using Microservices and Kubernetes

# data[].attributes.body
Name: Xing Huang
Student ID:18004668

# data[].attributes.headline
Developing applications using Microservices and Kubernetes

# data[].attributes.subject
Developing applications using Microservices and Kubernetes

# data[].attributes.body
Name: Tommy Yoon
Student ID: 20104020

# data[].attributes.headline
Developing Chatbots and its future trends

# data[].attributes.subject
Developing Chatbots and its future trends

# data[].attributes.body
Name  : Mehreen Manzoor
Student ID : 19086093

# data[].attributes.body
Name: Yatin Grover
Student ID: 20105274

# data[].attributes.headline
Developing applications using Deep learning

# data[].attributes.subject
Developing applications using Deep learning

# data[].attributes.body
Name  :Sri Nidhi Hrushi Kesav Regula
Student ID : 14855245

# data[].attributes.headline
Using image recognition in applications ( face recognition)

# data[].attributes.subject
Using image recognition in applications ( face recognition)

# data[].attributes.body
Name: Kaihan Wang
Student ID: 15907498

# data[].attributes.headline
Developing smart devices and security applications using IOT

# data[].attributes.subject
Developing smart devices and security applications using IOT

# data[].attributes.body
Name :- Ravindran G Nair (Ravi)
Student ID :- 20091988

# data[].attributes.body
Name: Shuyao Cai
StudentID: 19065388

# data[].attributes.body
Name :- Sherin Jacob
Student ID :- 16945505

# data[].attributes.headline
Developing applications using Deep learning

# data[].attributes.subject
Developing applications using Deep learning

# data[].attributes.body
Name: Nikhil Anil Sneha
Student ID: 16945481

# data[].attributes.body
Name: Zaw Lin Tun
Student ID: 16925410

# data[].attributes.headline
Developing applications using Deep Learning

# data[].attributes.subject
Developing applications using Deep Learning

# data[].attributes.body
Name : Brian De Silva
Student ID : 16954670

# data[].attributes.body
Name: Jesphilmar Iyog
Student ID: 15920323

# data[].attributes.headline
Developing applications using Deep Learning

# data[].attributes.subject
Developing applications using Deep Learning

# data[].attributes.body
Name:Naveen De Silva
Student ID: 16944852

# data[].attributes.body
Name: Jitendra Singh Kushwaha
Student ID: 20093893

# data[].attributes.body
Name: Johnny He
Student ID: 16939881

# data[].attributes.headline
Developing applications using the Internet of Things

# data[].attributes.subject
Developing applications using the Internet of Things

# data[].attributes.body
Name: Yin Zhang
Student ID: 20089468

# data[].attributes.headline
Developing applications using Microservices and Kubernetes

# data[].attributes.subject
Developing applications using Microservices and Kubernetes

# data[].attributes.body
Name: Jialin Zang
Student ID: 200080238

# data[].attributes.headline
Developing chatbots by using Google DialogFlow

# data[].attributes.subject
Developing chatbots by using Google DialogFlow

# data[].attributes.body
Name: Purva Arte
Student ID: 19054377

# data[].attributes.headline
Developing applications using Microservices and Kubernetes

# data[].attributes.subject
Developing applications using Microservices and Kubernetes

# data[].attributes.body
Name: Alex Kim
Student ID: 1397025

# data[].attributes.body
Name: Nishant Patel
Student ID: 16945747

# data[].attributes.headline
Developing applications using Deep Learning

# data[].attributes.subject
Developing applications using Deep Learning

# data[].attributes.body
Name: Xuetong Ji
Student ID: 16901042

# data[].attributes.headline
Developing applications using the Internet of Things

# data[].attributes.subject
Developing applications using the Internet of Things

# data[].attributes.body
Name: Dongchen Duan
Student ID: 20092618

# data[].attributes.body
Student ID: 19077174
Name: Karthick Sundar Subramanian

# data[].attributes.body
Name: Kai Liu
Student ID :20091110

# data[].attributes.body
Student ID:19082200
Name: Hang Yi Chung

# data[].attributes.headline
Using a DevOps way of working including feature flags, post deployment monitoring

# data[].attributes.subject
Using a DevOps way of working including feature flags, post deployment monitoring

# data[].attributes.body
Name: Sebin Benjamin
Student ID : 19069073

# data[].attributes.body
Name: Suby Siby
Student ID: 20087421