# data[].attributes.body
AWS lanza Amazon Kinesis Data Streams bajo demanda (infoq.com)

# data[].attributes.attachment_link.readable_text
Amazon Kinesis Data Streams is a fully-managed, serverless service on AWS for real-time processing of streamed data at a massive scale. Recently, the company released a new capacity mode On-demand for the service, which eliminates capacity provisioning and management for streaming workloads.

With Amazon Kinesis Data Streams, customers can take any amount of data from any number of sources and scale up and down as needed. Earlier, they needed to adjust and define the capacity of the service. Now scaling with the "on-demand capacity" mode is done automatically in response to varying data traffic.

In a recent AWS news blog post on Amazon Kinesis Data Streams, Marcia Villalba, a senior developer advocate for Amazon Web Services, explains when customers can benefit from the On-demand mode:

On-demand mode is great for customers that have an unknown or variable workload, or who simply don't want to deal with capacity management. On-demand mode works best for workloads that have even partition key distribution. For example, you run a mobile game that has variable traffic through the week or day, as customers play mostly on nights or weekends. Or, you run a streaming platform that hosts live shows, and you see a sudden increase in demand depending on the guests you have.

Users of Amazon Kinesis Data Streams can create a new data stream with Kinesis Data Streams On-demand from the AWS console, and through AWS SDKs, AWS Command Line Interface (CLI), and AWS CloudFormation. When using the console, the capacity mode can be selected in the Create data stream section.

Source: https://aws.amazon.com/blogs/aws/amazon-kinesis-data-streams-on-demand-stream-data-at-scale-without-managing-capacity/

Adding the On-demand mode capacity mode for Amazon Kinesis Data Streams is a response by AWS to its competitors, which offer similar services that also auto-scale. For instance, Azure Event Hubs Microsoft's streaming service had the auto-inflate feature for some time now responsible for automatic scaling of capacity. And Google's Pub/Sub streaming service also has auto-scaling and provisioning features.

James Urquhart, global field CTO at VMware, stated in a tweet:

This is the purchasing experience I expect all flow and event processing/queueing services to adopt over time. Pay for throughput.

And Clemens Vasters, principal architect messaging at Microsoft, responded to that a tweet:

Azure Event Hubs Standard and Premium already had auto-scale of Throughout/Processing Units independent of how many partitions you set up. Eventually, everyone gets there.

Currently, Amazon Kinesis Data Streams On-demand is available globally in all commercial regions. Customers are charged per gigabyte of data written, read, and stored in the stream, in a pay-per-throughput fashion - more details are on pricing can be found on the pricing page.

# data[].attributes.attachment_link.title
AWS Launches Amazon Kinesis Data Streams On-Demand

# data[].attributes.attachment_link.description
Amazon Kinesis Data Streams is a fully-managed, serverless service on AWS for real-time processing of streamed data at a massive scale. Recently, the company released a new capacity mode On-demand for the service, which eliminates capacity provisioning and management for streaming workloads.

# data[].attributes.attachment_link.title
Comparación de AWS con los servicios de Azure - Azure Architecture Center

# data[].attributes.attachment_link.description
Compare Azure Cloud Services con Amazon Web Services (AWS) para soluciones de nube múltiple o migración a Azure.

# data[].attributes.body
Basado en arquitectura Apache Kafka

[API, web scraping - social network - telemetry] -
source client -
productor -
tópico -
consumidor -

procesamiento

# data[].attributes.attachment_link.readable_text
The rise of distributed log technologies

Distributed log technologies such as Apache Kafka, Amazon Kinesis, Microsoft Event Hubs and Google Pub/Sub have matured in the last few years, and have added some great new types of solutions when moving data around for certain use cases.

According to IT Jobs Watch, job vacancies for projects with Apache Kafka have increased by 112% since last year, whereas more traditional point to point brokers haven't faired so well. Jobs advertised with Active MQ have decreased by 43% compared to this time last year. Rabbit MQ is a more modern version of the traditional message broker and efficiently implements AMQP but only saw an increase of 22% in terms of job adverts which ask for it. I suspect in reality a lot of the use cases that were poorly served by traditional message brokers have moved very quickly to distributed log technologies, and a lot of new development that traditionally used technologies such as Active MQ have moved to Rabbit MQ.

What are they?

While distributed log technologies may on the surface seem very similar to traditional broker messaging technologies, they differ significantly architecturally and therefore have very different performance and behavioural characteristics.

Traditional message broker systems such as those which are JMS or AMQP compliant tend to have processes which connect direct to brokers, and brokers which connect direct to processes. Hence for a message to go from one process to another, it can do so routed via a broker. These solutions tend to be optimised towards flexibility and configurable delivery guarantees.

Recently technologies such as Apache Kafka and the like, or as they are commonly called the distributed commit log technologies have come along and can play a similar role to the traditional broker message systems. They are optimised towards different use cases however, instead of concentrating on flexibility and delivery guarantees, they tend to be concentrated on scalability and throughput.

In a distributed commit log architecture the sending and receiving processes are a bit more de-coupled and in some ways the sending process doesn't care about the receiving processes. The messages are persisted immediately to the distributed commit log. The delivery guarantees are often viewed in the context that messages in the distributed commit log tend to be persisted only for a period of time. Once the time has expired, older messages in the distributed commit log disappear regardless of guarantees and therefore usually fit use cases where the data can expire, or will be processed by a certain time.

Choice of distributed commit log technologies

In this article we will compare the four most popular distributed log technologies. Here is a high-level summary of each:

Apache Kafka

Kafka is a distributed streaming service originally developed by LinkedIn. APIs allow producers to publish data streams to topics. A topic is a partitioned log of records with each partition being ordered and immutable. Consumers can subscribe to topics. Kafka can run on a cluster of brokers with partitions split across cluster nodes. As a result, Kafka aims to be highly scalable. However, Kafka can require extra effort by the user to configure and scale according to requirements.

Amazon Kinesis

Kinesis is a cloud based real-time processing service. Kinesis producers can push data as soon as it is created to the stream. Kenesis breaks the stream across shards (similar to partitions), determined by your partition key. Each shard has a hard limit on the number of transactions and data volume per second. If you exceed this limit, you need to increase your number of shards. Much of the maintenance and configuration is hidden from the user. AWS allows ease of scaling with users only paying for what they use.

Microsoft Azure Event Hubs

Event Hubs describes itself as an event ingestor capable of receiving and processing millions of events per second. Producers send events to an event hub via AMQP or HTTPS. Event Hubs also have the concept of partitions to enable specific consumers to receive a subset of the stream. Consumers connect via AMQP. Consumer groups are used to allow consuming applications to have a separate view of the event stream. Event Hubs is a fully managed service but users must pre-purchase capacity in terms of throughput units.

Google Pub/Sub

Pub/Sub offers scalable cloud based messaging. Publisher applications send messages to a topic with consumers subscribing to a topic. Messages are persisted in a message store until they are acknowledged. Publishers and pull-subscribers are applications that can make Google API HTTPS requests. Scaling is automatic by distributing load across data centres. Users are charged by data volume.

It's not possible in this article to examine each technology in great detail. However, as an example we will explore Apache Kafka more within the following section.

Kafka Architecture

The architecture around Kafka is comprised of the following components:

Topics - this is a conceptual division of grouped messages. It could be stock prices, or cars seen, or whatever.

Partitions - this is how parallelism is easily achieved. A topic can be split into 1 or more partitions. Then each message is kept in an ordered queue within that partition (messages are not ordered across partitions).

Consumers - 0, 1 or more consumers can process any partition of a topic.

Consumer Groups - these are groups of consumers that are used to load share. If a consumer group is consuming messages from one partition, each consumer in a consumer group will consume a different message. Consumer groups are typically used to load share.

Replication - you can set the replication factor on Kafka on a per topic basis. This will can help reliability if one or more servers fail.

What are distributed commit log technologies good for?

While I won't compare and contrast all the use cases of traditional message brokers, when compared with distributed log technologies (we'll save that for another blog) but if for a moment we compare the drivers behind the design of Active MQ and Apache Kafka we can get a feel for what they are good for. Apache Kafka was built in Scala at LinkedIn to provide a way to scale out their updates and maximise throughput. Scalability was of paramount concern over latency and message delivery guarantees. Within the scalability requirement was the need to simplify configuration, management and monitoring.

There are a few use cases that distributed log technologies really excel at, which often have the following characteristics:

The data has a natural expiry time - such as the price of a stock or share.

The data is of a large volume - therefore throughput and scalability are key considerations.

The data is a natural stream - there can be value in going back to certain points in the stream or traversing forward to a given point.

So certainly within financial services, Kafka is used for:

Price feeds

Event Sourcing

Feeding data into a data lake from OLTP systems

Attributes of distributed log technologies

There are a number of attributes that should be considered when choosing a distributed log technology, these include:

Messaging guarantee

Ordering guarantee

Throughput

Latency

Configurable persistence period

Persisted storage

Partitioning

Consumer Groups

Messaging guarantee

Messaging systems typically provide a specific guarantee when delivering messages. Some guarantees are configurable. The types of guarantees are:

At most once - some messages may be lost, and no message is delivered more than once.

Precisely once - each message is guaranteed to be delivered once only, not more or less.

At least once - each message is guaranteed to be delivered, but may in some cases be delivered multiple times.

Ordering guarantee

For distributed log technologies the following ordering guarantees are possible

None - there is absolutely no guarantee of any messages coming out in any order related to the order they came in.

Within a partition - within any given partition the order is absolutely guaranteed, but across partitions the messages may be out of order.

Across partitions - ordering is guaranteed across partitions. This is very expensive and slows down and makes scaling a lot more complicated.

Throughput

The volume of messages that can be processed within a set period of time.

Latency

The average speed a message is processed after it is put on the queue. It is worth noting that you can sometimes sacrifice latency to get throughput by batching things together (conversely, improving latency by sending data immediately after it is created can sacrifice throughput).

Configurable persistence period

The period of time that messages will be kept for. Once that time has passed the message is deleted, even if no consumers have consumed that message.

How to choose which one to use?

When choosing between the distributed commit log technologies there are a few big questions you can ask yourself (captured in a simple decision flow chart below). Are you looking for a hosted solution or a managed service solution? While Kafka is possibly the most flexible of all the technologies, it is also the most complex to operate and manage. What you save in the cost of the tool, you will probably use up in support and devops running and managing.

Common to all distributed commit log technologies is that the messaging guarantees tends to be at least once processing. This means the consumer needs to protect against receiving the message multiple times. Combining with a stream processing engine such as Apache Spark can give the consumer precisely once processing and remove the need for the user application to handle duplicate messages.

Kafka

Amazon Kinesis

Microsoft Azure Event Hubs

Google pub/sub

Messaging guarantees

At least once per normal connector.
Precisely once with Spark direct Connector.

At least once unless you build deduping or idempotency into the consumers.

At least once but allows consumer managed checkpoints for exactly once reads.
At least once

Ordering guarantees

Guaranteed within a partition.

Guaranteed within a shard.

Guaranteed within partition.

No ordering guarantees.

Throughput

No quoted throughput figures. Study showed a throughput of ~30,000 messages/sec.

One shard can support 1 MB/s input, 2 MB/s output or 1000 records per second. Study showed a throughput of ~20,000 messages/sec.

Scaled in throughput units. Each supporting 1 MB/s ingress, 2 MB/s egress or 84 GB storage. Standard tier allows 20 throughput units.

Default is 100MB/s in, 200MB/s out but maximum is quoted as unlimited.

Partitioning

Yes

Yes (Shards)

Yes

Yes - but not under user control

Consumer groups

Yes

Yes (called auto-scaling groups)

Yes (up to 20 for the standard pricing tier)

Yes (called subscriptions)

Disaster recovery - with across region replication

Yes (cluster mirroring)

Automatically across 3 zones

Yes (for the standard tier)

Yes (automatic)

Change partitioning after setup

Yes (increase only - does not re-partition existing data)

Yes by "resharding" (merge or split shards).

No (not under user control)

Partition/shard limit

No limit. Optimal partitions depends on your use case.

500 (US/EU) or 200 (other) although you can apply to Amazon to increase this.

Between 28 and 32 (can pay for more).

Not visible to user.

Latency

Milliseconds for some set-ups. Benchmarking showed ~2 ms median latency.

No quoted figures.

No quoted figures.

Replication

Configurable replicas. Acknowledgement of message published can be on send, on receipt or on successful replication (local only).

Hidden (across three zones). Message published acknowledgement is always after replication.

Configurable (and allowed across regions for the standard tier).

Hidden. Message published acknowledgement after half the disks on half the clusters have the message.

Languages supported

Java, Go, Scala, Clojure, python, C++, .NET, .NET Core, Node.js, PHP, Python, Ruby, Spark and many more...

Java, Go, .NET, .NET Core, Node.js, PHP, Python, Ruby, Spark. All the API's for pub/sub are in currently beta with the exception of the PHP API which is at General Availability stage.

Decision flow chart

To help you choose here is a decision flow chart. It certainly isn't recommended you stick to it rigidly. It's more a general guide to which technologies to consider, and a few decision points to help you eliminate some technologies, so you have a more focused pool to evaluate/compare. Ultimately Kafka is the most flexible, and has great performance characteristics. But it is also requires the most energy - both in setup and monitoring/maintaining. If you wish to go more for a fully managed solution - Kinesis, Event Hubs and pub/sub offer alternative options depending on whether ordering and blob size are important to you.

Hopefully this blog post will help you choose the technology that is right for you, and I am very interested if you have chosen one recently and what reason you chose it over the others.

References

Technology Vacancies Statistics

The following statistics were taken from IT Jobs Watch, which highlights the changing number of vacancies for each technology between 2016 and 2018.

# data[].attributes.attachment_link.title
Comparing Apache Kafka, Amazon Kinesis, Microsoft Event Hubs and Google Pub/Sub

# data[].attributes.attachment_link.description
Distributed log technologies such as Apache Kafka, Amazon Kinesis, Microsoft Event Hubs and Google Pub/Sub have matured in the last few years, and have added some great new types of solutions when moving data around for certain use cases.

# data[].attributes.body
Investigar y realizar una comparacion entre los principales servicios de Streaming que nos ofrecen los 3 vendros Cloud.

# data[].attributes.attachment_link.title
Enterprise IT Software Reviews | Gartner Peer Insights

# data[].attributes.attachment_link.description
Enterprise IT Software Reviews | Gartner Peer Insights