# data[].attributes.headline
Configure multiple SSL certificates in Haproxy - Server Fault

# data[].attributes.subject
Configure multiple SSL certificates in Haproxy - Server Fault

# data[].attributes.body
I used this to configure HAPROXY so that it can use a *.padlet.org certificate and a padlet.com certificate.

# data[].attributes.attachment_link.readable_text
Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up

Here's how it works:

1. Anybody can ask a question

2. Anybody can answer

3. The best answers are voted up and rise to the top

My haproxy instance serves 2 domains (mostly to avoid XSS on the main site).

The rules look something like this

bind :443 ssl crt /etc/ssl/haproxy.pem acl is_static hdr_end(Host) -i example.com acl is_api hdr_end(Host) -i api.example.com acl is_files hdr_end(Host) -i example.io redirect scheme https if !{ ssl_fc } is_static is_api

Now SSL uses /etc/ssl/haproxy.pem as the default cert, which is the certificate for example.com and not example.io.

How can I specify certs for multiple domain names?

asked Dec 12 '13 at 17:37

You can concatenate all your certificates into files say haproxy1.pem and haproxy2.pem or you can specify a directory containing all your pem files.

cat cert1.pem key1.pem > haproxy1.pem cat cert2.pem key2.pem > haproxy2.pem

As per the haproxy docs

Then on the config use something like this:

defaults log 127.0.0.1 local0 option tcplog frontend ft_test mode http bind 0.0.0.0:443 ssl crt /certs/haproxy1.pem crt /certs/haproxy2.pem use_backend bk_cert1 if { ssl_fc_sni my.example.com } # content switching based on SNI use_backend bk_cert2 if { ssl_fc_sni my.example.org } # content switching based on SNI backend bk_cert1 mode http server srv1 <ip-address2>:80 backend bk_cert2 mode http server srv2 <ip-address3>:80

Read more about SNI

Keep in mind that SSL support is in development staging for haproxy and also that it apparently has considerable performance hit.

There are other solutions talked about in this thread: https://stackoverflow.com/questions/10684484/haproxy-with-multiple-https-sites

Hope this helps.

answered Dec 12 '13 at 18:19

Not the answer you're looking for? Browse other questions tagged haproxy or ask your own question.

Server Fault works best with JavaScript enabled

# data[].attributes.attachment_link.title
Configure multiple SSL certificates in Haproxy

# data[].attributes.attachment_link.description
My haproxy instance serves 2 domains (mostly to avoid XSS on the main site). The rules look something like this bind :443 ssl crt /etc/ssl/haproxy.pem acl is_static hdr_end(Host) -i example.co...

# data[].attributes.headline
Using PostgreSQL Arrays The Right Way | Heap Blog

# data[].attributes.subject
Using PostgreSQL Arrays The Right Way | Heap Blog

# data[].attributes.attachment_link.readable_text
At Heap, we lean on PostgreSQL for most of the backend heavy lifting. We store each event as an hstore blob, and we keep a PostgreSQL array of events done by each user we track, sorted by time. Hstore lets us attach properties to events in a flexible way, and arrays of events give us great performance, especially for funnel queries, in which we compute the drop off between different steps in a conversion funnel. In this post, we'll take a look at a PostgreSQL function that unexpectedly hung on large inputs and rewrite it in an efficient, idiomatic way.
Your first instinct might be to treat arrays in PostgreSQL like their analogues in C-based languages. You might be used to manipulating data via array positions or slices. Be careful not to think this way in PostgreSQL, especially if your array type is variable length, e.g. json, text, or hstore. If you're accessing a PostgreSQL array via its positions, you're in for an unexpected performance blowup.

This came up a few weeks ago at Heap. We keep an array of events for each user tracked by Heap, in which we represent each event with an hstore datum. We have an import pipeline that appends new events to the right arrays. In order to make this import pipeline idempotent, we give each event an event_id entry, and we run our event arrays through a utility function that squashes duplicates. If we want to update the properties attached to an event, we just dump a new event into the pipeline with the same event_id.

So, we need a utility function that takes an array of hstores, and, if two events have the same event_id, it should take the one that occurs later in the array. An initial attempt to write this function looked like this:

This works, but it blows up for large inputs. It's quadratic, and it takes almost 40 seconds for an input array of 100k elements!

What's going on here? The issue is that PostgreSQL stores an array of hstores as an array of values, not an array of pointers to values. That is, an array of three hstores looks something like
{"event_id=>1,data=>foo", "event_id=>2,data=>bar", "event_id=>3,data=>baz"}
under the hood, as opposed to
{[pointer], [pointer], [pointer]}

For types that are variable length, e.g. hstores, json blobs, varchars, or text fields, PostgreSQL has to scan the array to find the Nth element. That is, to evaluate events[2], PostgreSQL parses events from the left until it hits the second entry. Then, for events[3], it re-scans from the first index all over again until it hits the third entry! So, evaluating events[sub] is O(sub), and evaluating events[sub] for each index in the array is O(N ²), where N is the length of the array.

PostgreSQL could be smarter about caching intermediate parse results, or it could parse the array once in a context like this. The real answer is for arrays of variable-length elements to be implemented with pointers to values, so that we can always evaluate events[i] in constant time.

Even so, we shouldn't rely on PostgreSQL handling this well, since this is not an idiomatic query. Instead of generate_subscripts, we can use , which parses an array and returns a set of entries. This way, we never have to explicitly index into the array.

This is efficient, and it takes an amount of time that's linear in the size of the input array. It takes about a half a second for an input of 100k elements, compared to 40 seconds with the previous implementation.

This does what we want:
Lesson learned: if you find yourself accessing positions in a PostgreSQL array, consider unnest instead.
We like to avoid stubbing our toes. Have any feedback or other PostgreSQL tips? Shoot us a note @heap.

In particular, we use a lovely tool called Citus Data. More on that in another blog post!
See: https://heapanalytics.com/features/funnels. In particular, computing a conversion funnel requires a single scan over the array of events a user has done and doesn't require any joins.

# data[].attributes.attachment_link.title
Using PostgreSQL Arrays The Right Way - Heap | Mobile and Web Analytics

# data[].attributes.attachment_link.description
At Heap, we lean on PostgreSQL for most of the backend heavy lifting.[1] We store each event as an hstore blob, and we keep a PostgreSQL array of events done by each user we track, sorted by time. Hstore lets us attach properties to events in a flexible way, and arrays of events give us ...

# data[].attributes.headline
How to install gem from GitHub source? - Stack Overflow

# data[].attributes.subject
How to install gem from GitHub source? - Stack Overflow

# data[].attributes.attachment_link.description
I would like to install gem from the latest GitHub source. How do I do this?

# data[].attributes.body
Thus started a brief investigation into DNS resolvers -- DO shut Pioneer down because it was party to a DNS amplification attack.

# data[].attributes.attachment_link.description
Dnsmasq is a lightweight, easy to configure, DNS forwarder and DHCP server. It is designed to provide DNS and optionally, DHCP, to a small network. It can serve the names of local machines which are not in the global DNS.

# data[].attributes.headline
C0DE517E: Stuff that every programmer should know: Data Visualization

# data[].attributes.subject
C0DE517E: Stuff that every programmer should know: Data Visualization

# data[].attributes.attachment_link.readable_text
If you're a programmer and you don't have visualization as one of your main tools in your belt, then good news, you just found how to easily improve your skill set. Really it should be taught in any programming course.
Note: This post won't get you from zero to visualization expert, but hopefully it can pique your curiosity and will provide plenty of references for further study.

Visualizing data has two main advantages compared to looking at the same data in a tabular form.

The first is that we can pack more data in a graph that we can get by looking at numbers on screen, even more if we make our visualizations interactive, allowing explorations inside a data set. Our visual bandwidth is massive!

This is useful also because it means we can avoid (or rely less on) summarization techniques (statistics) that are always by their nature "lossy" and can easily hide important details (the Anscombe's quartet is the usual example).

The second advantage, which is even more important, is that we can reason about the data much better in a visual form.

0.2, 0.74, 0.99, 0.87, 0.42, -0.2, -0.74, -0.99, -0.87, -0.42, 0.2

What's that? How long do you have to think to recognize a sine in numbers? You might start reasoning about the simmetries, 0.2, -0.2, 0.74, -0.74, then the slope and so on, if you're very bright. But how long do you think it would take to recognize the sine plotting that data on a graph?

Yet I've rarely seen people dumping data from programs into graphing software and I've rarely seen programmers that actually know the science of data visualization.

Visualizing program behaviour is even more important in the context of rendering engineers or any code that doesn't just either fail hard or work right.

We can easily implement algorithms that are wrong but doesn't produce a completely broken output. It might be just slower (i.e. to converge) than it needs to be, or more noisy, or just not quite "right" and cause our artists to try to adjust for our mistakes by authoring fixes in the art (this happens -all- the time) and so on.

And there are even situations where the output is completely broken, but it's just not obvious from looking at a tabular output, a great example for this would be in the structure of LCG random numbers.

The main objective of visualization is to be meaningful. That means choosing the right data to study a problem, and displaying it in the right projection (graph, scale, axes...).

The right data is the one that is interesting, it shows the features of our problem. What questions are we answering (purpose)? What data we need to display?

The right projection is the one that shows such features in an unbiased, perceptually linear way, and that makes different dimensions comparable and possibly orthogonal. How do we reveal the knowledge that data is hiding? Is it x or 1/x? Log(x)? Should we study the ratio between quantities or absolute difference and so on.

Information about both data and scale comes at first from domain expertise. A light (or sound) intensity probably should go on a logarithmic scale, maybe a dot product should be displayed as the angle between its vectors, many quantities have a physical interpretation and a perceptual interpretation or a geometrical one and so on.

But even more interestingly, information about data can come from the data itself, by exploration. In an interactive environment it's easy to just dump a lot of data to observe, notice certain patterns and refine the graphs and data acquisition to "zoom in" particular aspects. Interactivity is the key (as -always- in programming).

When you delve a bit into visualization you'll find that there are two fairly distinct camps.

One is visualization of categorical data, often discrete, with the main goal of finding clusters and relationships.

The other camp is about visualization of continuos data, often in the context of scientific visualization, where we are interested in representing our quantities without distortion, in a way that the are perceptually linear.

This usually employs mostly position as a visual cue, thus 2d or 3d line/surface or point plots.

These become harder with the increase of dimensionality of our data as it's hard to go beyond three dimensions. Color intensity and "widgets" could be used to add a couple more dimensions to points in a 3d space but it's often easier to add dimensions by interactivity (i.e. slicing through the dataset by intersecting or projecting on a plane) instead.

Both kinds of visualizations have applications to programming. For deterministic processes, like the output or evolution in time of algorithms and functions, we want to monitor some data and represent it in an objective, undistorted manner. We know what the data means and how it should work, and we want to check that everything goes according to what we think it should.
But there are also times were we don't care about exact values but we seek for insight into processes of which we don't have exact mental models. This applies to all non-deterministic issues, networking, threading and so on, but also to many things that are deterministic in nature but have a complex behaviour, like memory hierarchy accesses and cache misses.

Whatever your visualization is though, t- Learn about perception caveats he first thing to be aware of is visual perception: n ot all visual cues are useful for quantitative analysis.

Perceptual biases are a big problem, because as they are perceptual, we tend not to see them, just subconsciously we are drawn to some data points more than others when we should not. Beware

of color, one of the most abused, misunderstood tool for quantitative data. Color (hue) is extremely hard to get right, it's very subjective and it doesn't express well quantities nor relationships (what color is less than another), y et it's used everywhere.

Intensity and saturation are not great either, again very commonly used but often inferior to other hints like point size or stroke width.

- Visualization of programs

Programs are these incredibly complicated projects we manage to carry forward, but if that's not challenging enough we really love working with them in the most complicated ways possible.

So of course visualization is really limited. The only "mainstream" usage you will have probably encountered is in the form of bad graphs of data from static analysis. Dependences, modules, relationships and so on.

Certainly if you have to see your program execution itself it -has- to be text. Watch windows, memory views with hex dumps and so on. Visual Studio, which is probably the best debugger IDE we have, is not visual at all nor allows for easy development of visualizations (it's even hard to grab data from memory in it).

We're programmers so it's not a huge deal to dump data to a file or peek memory [... my article], then we can visualize the output of our code with tools that are made for data.

But an even more important tool is to use visualization directly of the behaviour of code, in runtime. This is really a form of tracing which most often is limited to what's known as "printf" debugging.

Tracing is immensely powerful as it tells us at a high level what our code is doing, as opposed to the detailed inspection of how the code is running that we can get from stepping in a debugger.

Unfortunately there is today basically no tool for graphical representation of program state in time, so you'll Ideally though it would be lovely to be able to Apple's recent Microsoft had a have to roll your own. Working on your own sourcecode it's easy enough to put some instrument compiled code, it's definitely possible but much more of an hassle without the support of a debugger. Another alternative that sometimes I adopt is to just have an external application peek at regular interval into my target's process memory. instrumentation to export data to a Swift language seems a step into the right direction, and looks like it pulled some ideas from Bret Victor and timid plugin for VisualStudio that did some very basic plotting that doesn't seem to be actively updated and another one for in-memory images, but what would be really needed is the ability to export data easily and in realtime as good visualizations are usually to be made ad-hoc for a specific problem. Light Table. live graph and in my own experiments I don't use any library for this, just write the simplest possible ad-hoc code to suck the data out.

It's simple enough but it captures data at a very low frequency so it's not always applicable, I use it most of the times not on programs running in realtime but as an live memory visualization while stepping through in a debugger.

If you want to delve deeper into program visualization there is a fair bit written about it by the academia, with also a few interesting conferences, b ut what's even more interesting to me is seeing it applied to one of the hardest coding problems: reverse engineering.
It should perhaps not be surprising as reversers and hackers are very smart people, so it should be natural for them to use the best tools in their job.

It's quite amazing seeing how much one can understand with very little other information by just looking at visual fingerprints, data entropy and code execution patterns.
And again visualization is a process of exploration, it can highlight some patterns and anomalies to then delve in further with more visualizations or by using other tools.

Some of my other posts

d3.js could be nifty too, especially if you already have a web frontend for your application as it's increasingly popular to do

Don't use Excel, you'll waste time as it's very cumbersome to manipulate data there and its graphs are actually quite bad

Other Articles and websites (be sure to check the links -inside- the post first)

- Addendum: a random sampling of stuff I do for work
All made either in Mathematica or Processing and they are all interactive, realtime.

# data[].attributes.attachment_link.title
Stuff that every programmer should know: Data Visualization

# data[].attributes.attachment_link.description
If you're a programmer and you don't have visualization as one of your main tools in your belt, then good news, you just found how to easily improve your skill set. Really it should be taught in any programming course.

# data[].attributes.headline
How To Use HAProxy to Set Up HTTP Load Balancing on an Ubuntu VPS | DigitalOcean

# data[].attributes.subject
How To Use HAProxy to Set Up HTTP Load Balancing on an Ubuntu VPS | DigitalOcean

# data[].attributes.attachment_link.readable_text
About HAProxy

HAProxy(High Availability Proxy) is an open source load balancer which can load balance any TCP service. It is particularly suited for HTTP load balancing as it supports session persistence and layer 7 processing.

With the introduction of Shared Private Networking in DigitalOcean HAProxy can be configured as a front-end to load balance two VPS through private network connectivity.

Prelude

We will be using three VPS (droplets) here:

Droplet 1 - Load Balancer
Hostname: haproxy
OS: Ubuntu
Public IP: 1.1.1.1
Private IP: 10.0.0.100

Droplet 2 - Node 1
Hostname: lamp1
OS: LAMP on Ubuntu
Private IP: 10.0.0.1

Droplet 2 - Node 2
Hostname: lamp2
OS: LAMP on Ubuntu
Private IP: 10.0.0.2

Installing HAProxy

Use the apt-get command to install HAProxy.

apt-get install haproxy

We need to enable HAProxy to be started by the init script.

nano /etc/default/haproxy

Set the ENABLED option to 1

To check if this change is done properly execute the init script of HAProxy without any parameters. You should see the following.

root@haproxy:~# service haproxy Usage: /etc/init.d/haproxy {start|stop|reload|restart|status}

Configuring HAProxy

We'll move the default configuration file and create our own one.

mv /etc/haproxy/haproxy.cfg{,.original}

Create and edit a new configuration file:

nano /etc/haproxy/haproxy.cfg

Let us begin by adding configuration block by block to this file:

global log 127.0.0.1 local0 notice maxconn 2000 user haproxy group haproxy

The log directive mentions a syslog server to which log messages will be sent. On Ubuntu rsyslog is already installed and running but it doesn't listen on any IP address. We'll modify the config files of rsyslog later.

The maxconn directive specifies the number of concurrent connections on the frontend. The default value is 2000 and should be tuned according to your VPS' configuration.

The user and group directives changes the HAProxy process to the specified user/group. These shouldn't be changed.

defaults log global mode http option httplog option dontlognull retries 3 option redispatch timeout connect 5000 timeout client 10000 timeout server 10000

We're specifying default values in this section. The values to be modified are the various timeout directives. The connect option specifies the maximum time to wait for a connection attempt to a VPS to succeed.

The client and server timeouts apply when the client or server is expected to acknowledge or send data during the TCP process. HAProxy recommends setting the client and server timeouts to the same value.

The retries directive sets the number of retries to perform on a VPS after a connection failure.

The option redispatch enables session redistribution in case of connection failures. So session stickness is overriden if a VPS goes down.

listen appname 0.0.0.0:80 mode http stats enable stats uri /haproxy?stats stats realm Strictly\ Private stats auth A_Username:YourPassword stats auth Another_User:passwd balance roundrobin option httpclose option forwardfor server lamp1 10.0.0.1:80 check server lamp2 10.0.0.2:80 check

This contains configuration for both the frontend and backend. We are configuring HAProxy to listen on port 80 for appname which is just a name for identifying an application. The stats directives enable the connection statistics page and protects it with HTTP Basic authentication using the credentials specified by the stats auth directive.

This page can viewed with the URL mentioned in stats uri so in this case, it is http://1.1.1.1/haproxy?stats;
a demo of this page can be viewed here.

The balance directive specifies the load balancing algorithm to use. Options available are Round Robin ( roundrobin), Static Round Robin ( static-rr), Least Connections ( leastconn), Source ( source), URI ( uri) and URL parameter ( url_param).

Information about each algorithm can be obtained from the official documentation.

The server directive declares a backend server, the syntax is:

server <name> <address>[:port] [param*]

The name we mention here will appear in logs and alerts. There are many paratmeters supported by this directive and we'll be using the check and cookie parameters in this article. The check option enables health checks on the VPS otherwise, the VPS is
always considered available.

Once you're done configuring start the HAProxy service:

service haproxy start

Testing Load Balancing and Failover

To test this setup, create a PHP script on all your web servers (backend servers - LAMP1 and LAMP2 here).
/var/www/file.php
<?php header('Content-Type: text/plain'); echo "Server IP: ".$_SERVER['SERVER_ADDR']; echo "\nClient IP: ".$_SERVER['REMOTE_ADDR']; echo "\nX-Forwarded-for: ".$_SERVER['HTTP_X_FORWARDED_FOR']; ?>

Now we will use curl and request this file multiple times.

> curl http://1.1.1.1/file.php Server IP: 10.0.0.1 Client IP: 10.0.0.100 X-Forwarded-for: 117.213.X.X > curl http://1.1.1.1/file.php Server IP: 10.0.0.2 Client IP: 10.0.0.100 X-Forwarded-for: 117.213.X.X > curl http://1.1.1.1/file.php Server IP: 10.0.0.1 Client IP: 10.0.0.100 X-Forwarded-for: 117.213.X.X

Notice here how HAProxy alternatively toggled the connection between LAMP1 and LAMP2, this is how Round Robin works. The client IP we see here is the Private IP address of the load balancer and the X-Forwarded-For header is your IP.

To see how failover works, go to a web server and stop the service:

lamp1@haproxy:~#service apache2 stop

Send requests with curl again to see how things work.

Session Stickiness

If your web application serves dynamic content based on users' login sessions (which application doesn't), visitors will experience odd things due to continuous switching between VPS. Session stickiness ensures that a visitor sticks on to the VPS which served their first request. This is possible by tagging each backend server with a cookie.

We'll use the following PHP code to demonstrate how session stickiness works.
/var/www/session.php
<?php header('Content-Type: text/plain'); session_start(); if(!isset($_SESSION['visit'])) { echo "This is the first time you're visiting this server"; $_SESSION['visit'] = 0; } else echo "Your number of visits: ".$_SESSION['visit']; $_SESSION['visit']++; echo "\nServer IP: ".$_SERVER['SERVER_ADDR']; echo "\nClient IP: ".$_SERVER['REMOTE_ADDR']; echo "\nX-Forwarded-for: ".$_SERVER['HTTP_X_FORWARDED_FOR']."\n"; print_r($_COOKIE); ?>

This code creates a PHP session and displays the number of page views in a single session.

Cookie insert method

In this method, all responses from HAProxy to the client will contain a Set-Cookie: header with the name of a backend server as its cookie value. So going forward the client (web browser) will include this cookie with all its requests and HAProxy will forward the request to the right backend server based on the cookie value.

For this method, you will need to add the cookie directive and modify the server directives under listen

 cookie SRVNAME insert server lamp1 10.0.0.1:80 cookie S1 check server lamp2 10.0.0.2:80 cookie S2 check

This causes HAProxy to add a Set-Cookie: header with a cookie named SRVNAME having its value as S1 or S2 based on which backend answered the request. Once this is added restart the service:

service haproxy restart

and use curl to check how this works.

> curl -i http://1.1.1.1/session.php HTTP/1.1 200 OK Date: Tue, 24 Sep 2013 13:11:22 GMT Server: Apache/2.2.22 (Ubuntu) X-Powered-By: PHP/5.3.10-1ubuntu3.8 Set-Cookie: PHPSESSID=l9haakejnvnat7jtju64hmuab5; path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Vary: Accept-Encoding Content-Length: 143 Connection: close Content-Type: text/plain Set-Cookie: SRVNAME=S1; path=/ This is the first time you're visiting this server Server IP: 10.0.0.1 Client IP: 10.0.0.100 X-Forwarded-for: 117.213.X.X Array ( )

This is the first request we made and it was answered by LAMP1 as we can see from Set-Cookie: SRVNAME=S1; path=/. Now, to emulate what a web browser would do for the next request, we add these cookies to the request using the --cookie parameter of curl.

> curl -i http://1.1.1.1/session.php --cookie "PHPSESSID=l9haakejnvnat7jtju64hmuab5;SRVNAME=S1;" HTTP/1.1 200 OK Date: Tue, 24 Sep 2013 13:11:45 GMT Server: Apache/2.2.22 (Ubuntu) X-Powered-By: PHP/5.3.10-1ubuntu3.8 Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Vary: Accept-Encoding Content-Length: 183 Connection: close Content-Type: text/plain Your number of visits: 1 Server IP: 10.0.0.1 Client IP: 10.0.0.100 X-Forwarded-for: 117.213.87.127 Array ( [PHPSESSID] => l9haakejnvnat7jtju64hmuab5 [SRVNAME] => S1 ) > curl -i http://1.1.1.1/session.php --cookie "PHPSESSID=l9haakejnvnat7jtju64hmuab5;SRVNAME=S1;" HTTP/1.1 200 OK Date: Tue, 24 Sep 2013 13:11:45 GMT Server: Apache/2.2.22 (Ubuntu) X-Powered-By: PHP/5.3.10-1ubuntu3.8 Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Vary: Accept-Encoding Content-Length: 183 Connection: close Content-Type: text/plain Your number of visits: 2 Server IP: 10.0.0.1 Client IP: 10.0.0.100 X-Forwarded-for: 117.213.87.127 Array ( [PHPSESSID] => l9haakejnvnat7jtju64hmuab5 [SRVNAME] => S1 )

Both of these requests were served by LAMP1 and the session was properly maintained. This method is useful if you want stickiness for all files on the web server.

Cookie Prefix Method

On the other hand, if you want stickiness only for specific cookies or don't want to have a separate cookie for session stickiness, the prefix option is for you.

To use this method, use the following cookie directive:

cookie PHPSESSID prefix

The PHPSESSID can be replaced with your own cookie name. The server directive remains the same as the previous configuration.

Now let's see how this works.

> curl -i http://1.1.1.1/session.php HTTP/1.1 200 OK Date: Tue, 24 Sep 2013 13:36:27 GMT Server: Apache/2.2.22 (Ubuntu) X-Powered-By: PHP/5.3.10-1ubuntu3.8 Set-Cookie: PHPSESSID=S1~6l2pou1iqea4mnhenhkm787o56; path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Vary: Accept-Encoding Content-Length: 143 Content-Type: text/plain This is the first time you're visiting this server Server IP: 10.0.0.1 Client IP: 10.0.0.100 X-Forwarded-for: 117.213.X.X Array ( )

Notice how the server cookie S1 is prefixed to the session cookie. Now, let's send two more requests with this cookie.

> curl -i http://1.1.1.1/session.php --cookie "PHPSESSID=S1~6l2pou1iqea4mnhenhkm787o56;" HTTP/1.1 200 OK Date: Tue, 24 Sep 2013 13:36:45 GMT Server: Apache/2.2.22 (Ubuntu) X-Powered-By: PHP/5.3.10-1ubuntu3.8 Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Vary: Accept-Encoding Content-Length: 163 Content-Type: text/plain Your number of visits: 1 Server IP: 10.0.0.1 Client IP: 10.0.0.100 X-Forwarded-for: 117.213.X.X Array ( [PHPSESSID] => 6l2pou1iqea4mnhenhkm787o56 ) > curl -i http://1.1.1.1/session.php --cookie "PHPSESSID=S1~6l2pou1iqea4mnhenhkm787o56;" HTTP/1.1 200 OK Date: Tue, 24 Sep 2013 13:36:54 GMT Server: Apache/2.2.22 (Ubuntu) X-Powered-By: PHP/5.3.10-1ubuntu3.8 Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Vary: Accept-Encoding Content-Length: 163 Content-Type: text/plain Your number of visits: 2 Server IP: 10.0.0.1 Client IP: 10.0.0.100 X-Forwarded-for: 117.213.X.X Array ( [PHPSESSID] => 6l2pou1iqea4mnhenhkm787o56 )

We can clearly see that both the requests were served by LAMP1 and the session is perfectly working.

Configure Logging for HAProxy

When we began configuring HAProxy, we added a line: log 127.0.0.1 local0 notice which sends syslog messages to the localhost IP address. But by default, rsyslog on Ubuntu doesn't listen on any address. So we have to make it do so.

Edit the config file of rsyslog.

nano /etc/rsyslog.conf

Add/Edit/Uncomment the following lines:

$ModLoad imudp $UDPServerAddress 127.0.0.1 $UDPServerRun 514

Now rsyslog will work on UDP port 514 on address 127.0.0.1 but all HAProxy messages will go to /var/log/syslog so we have to separate them.

Create a rule for HAProxy logs.

nano /etc/rsyslog.d/haproxy.conf

Add the following line to it.

if ($programname == 'haproxy') then -/var/log/haproxy.log

Now restart the rsyslog service:

service rsyslog restart

This writes all HAProxy messages and access logs to /var/log/haproxy.log.

Keepalives in HAProxy

Under the listen directive, we used option httpclose which adds a Connection: close header. This tells the client (web browser) to close a connection after a response is received.

If you want to enable keep-alives on HAProxy, replace the option httpclose line with:

option http-server-close timeout http-keep-alive 3000

Set the keep-alive timeout wisely so that a few connections don't drain all the resources of the load balancer.

Testing Keepalives

Keepalives can be tested using curl by sending multiple requests at the same time. Unnecessary output will be omitted in the following example:

> curl -v http://1.1.1.1/index.html http://1.1.1.1/index.html * About to connect() to 1.1.1.1 port 80 (#0) * Trying 1.1.1.1... connected > GET /index.html HTTP/1.1 > User-Agent: curl/7.23.1 (x86_64-pc-win32) libcurl/7.23.1 OpenSSL/0.9.8r zlib/1.2.5 > Host: 1.1.1.1 > Accept: */* > ......[Output omitted]......... * Connection #0 to host 1.1.1.1 left intact * Re-using existing connection! (#0) with host 1.1.1.1 * Connected to 1.1.1.1 (1.1.1.1) port 80 (#0) > GET /index.html HTTP/1.1 > User-Agent: curl/7.23.1 (x86_64-pc-win32) libcurl/7.23.1 OpenSSL/0.9.8r zlib/1.2.5 > Host: 1.1.1.1 > Accept: */* > .......[Output Omitted]......... * Connection #0 to host 1.1.1.1 left intact * Closing connection #0

In this output, we have to look for the line: Re-using existing connection! (#0) with host 1.1.1.1, which indicates that curl used the same connection to make subsequent requests.

# data[].attributes.attachment_link.title
How To Use HAProxy to Set Up HTTP Load Balancing on an Ubuntu VPS | DigitalOcean

# data[].attributes.attachment_link.description
HAProxy(High Availability Proxy) is an open source load balancer which can load balance any TCP service. It is particularly suited for HTTP load balancing as it supports session persistence and layer 7 processing. With the introduction of Shared Private Networking in DigitalOcean HAProxy can be configured as a front-end to load balance two VPS through private network connectivity.

# data[].attributes.headline
ddos - How can I identify open DNS resolvers in my network? - Information Security Stack Exchange

# data[].attributes.subject
ddos - How can I identify open DNS resolvers in my network? - Information Security Stack Exchange

# data[].attributes.attachment_link.readable_text
Identify an Open DNS server by your own querying via NMAP:

x.x.x.x = DNS server IP
nmap -sU -p 53 -sV -P0 --script "dns-recursion" x.x.x.x
Possible output would be:

PORT STATE SERVICE VERSION
53/udp open domain ISC BIND "version"
*|_dns-recursion: Recursion appears to be enabled*

If you prefer make use of online services, openresolver project is very good, it checks also subnets of /22 width, so check it out ---> http://www.openresolverproject.org

After an Open DNS server discovery with online tools is a good idea to do a double check getting a proof about recursion ---> http://www.kloth.net/services/dig.php

Example output, watch the "ra" flag means recursion available:
; <<>> DiG 9.7.3 <<>> @x.x.x.x domain.cn A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: xxx
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 5, ADDITIONAL: 0

DISABLING RECURSION
( source knowledgelayer softlayer com)

Disable Recursion in Windows Server 2003 and 2008

Access the DNS Manager from the Start menu: Click the Start button. Select Administrative Tools. Select DNS. Right click on the desired DNS Server in the Console Tree. Select the Proprerties tab. Click the Advanced button in the Server Options section. Select the Disable Recursion checkbox. Click the OK button.

Disable Recursion in Linux

Locate the BIND configuration file within the operating system. The BIND configuration file is usually located in one of the following paths: /etc/bind/named.conf /etc/named.conf Open the named.conf file in your preferred editor. Add the following details to the Options section: allow-transfer {"none";}; allow-recursion {"none";}; recursion no; Restart the device.

# data[].attributes.attachment_link.title
How can I identify open DNS resolvers in my network?

# data[].attributes.attachment_link.description
Identify an Open DNS server by your own querying via NMAP: x.x.x.x = DNS server IP nmap -sU -p 53 -sV -P0 --script "dns-recursion" x.x.x.x Possible output would be: PORT STATE SERVICE VERSION 53/udp open domain ISC BIND "version" *|_dns-recursion: Recursion appears to be enabled* If you prefer make use

# data[].attributes.headline
[ubuntu] 12.04 running DNSMASQ by default

# data[].attributes.subject
[ubuntu] 12.04 running DNSMASQ by default

# data[].attributes.attachment_link.title
[ubuntu] 12.04 running DNSMASQ by default

# data[].attributes.attachment_link.description
in 12.04 dnsmasq is now running by default due to being hardcoded into network manager. Potentially bad idea ;-) (ever heard of DNS cache poisoning or MITM attacks ?) 12.04 Release Notes 5th Bullet Point. Its built into Network manager now.

# data[].attributes.attachment_link.description
Learn how to get the most out of Swiftype

# data[].attributes.headline
Safari Extensions Development Guide: Using Extension Builder

# data[].attributes.subject
Safari Extensions Development Guide: Using Extension Builder

# data[].attributes.attachment_link.description
Explains how to create extensions to the Safari browser.

# data[].attributes.body
OK i'm going to defend JSON fetching results here. In this case, no indexes were created for the JSON type (not sure why). We should test querying results with the JSON type before passing judgement too quickly.

# data[].attributes.headline
postgresql - How to create index on json field in Postgres 9.3 - Stack Overflow

# data[].attributes.subject
postgresql - How to create index on json field in Postgres 9.3 - Stack Overflow

# data[].attributes.body
Creating indexes on JSON types.

# data[].attributes.attachment_link.title
How to create index on json field in Postgres 9.3

# data[].attributes.attachment_link.description
In PostgreSQL 9.3 Beta 2 (?), how do I create an index on a JSON field? I tried it using the -> operator used for hstore but got the following error: CREATE TABLE publishers(id INT, info JSON);

# data[].attributes.headline
What can you do with PostgreSQL and JSON? | clarkdave.net

# data[].attributes.subject
What can you do with PostgreSQL and JSON? | clarkdave.net

# data[].attributes.body
Using JSON for data warehousing on Amazon RDS is definitely an option.

# data[].attributes.attachment_link.readable_text
What can you do with PostgreSQL and JSON?

PostgreSQL 9.2 added a native JSON data type, but didn't add much else. You've got three options if you actually want to do something with it:

1. Wait for PostgreSQL 9.3 (or use the beta)

2. Use the plv8 extension. Valid option, but more DIY (you'll have to define your own functions)

3. Use the json_enhancements extension, which backports the new JSON functionality in 9.3 to 9.2

I wanted to use this stuff now, and I opted to go with option 3. I wrote a blog post which should help you get going if you want to go this route: adding json_enhancements to PostgreSQL 9.2.

So let's assume you're on either 9.3, or 9.2 with json_enhancements. What can you do? Lots! All the new JSON operators and functions are in the 9.3 documentation, so I'm going to run through some of the more fun things you can do along with a real-world use case.

Get started

Create a database to play about in:

createdb json_test psql json_test

With some sample data:

Selecting

You can use the JSON operators to pull values out of JSON columns:

The -> operator returns the original JSON type (which might be an object), whereas ->> returns text. You can use the -> to return a nested object and thus chain the operators:

How cool is that?

Filtering

Of course, you can also select rows based on a value inside your JSON:

You can also find rows based on the value of a nested JSON object:

Indexing

You can add indexes on any of these using PostgreSQL's expression indexes, which means you can even add unique constraints based on your nested JSON data:

Expression indexes are somewhat expensive to create, but once in place will make querying on any JSON property very fast.

A real world example

OK, let's give this a go with a real life use case. Let's say we're tracking analytics, so we have an events table:

We're going to store events in this table, like pageviews. Each event has properties, which could be anything (e.g. current page) and also sends information about the browser (like OS, screen resolution, etc). Both of these are completely free form and could change over time (as we think of extra stuff to track).

Let's insert a couple of events:

Hm, this is starting to remind me of MongoDB!

Collect some stats

Using the JSON operators, combined with traditional PostgreSQL aggregate functions, we can pull out whatever we want. You have the full might of an RDBMS at your disposal.
Browser usage? Total revenue per visitor? Average screen resolution?
You've probably got the idea, so I'll leave it here.

Is this better than MongoDB?

Haha, just kidding. I'm not going to answer that!

There's a whole lot of other JSON operators and functions I didn't cover here, for example to work with JSON arrays too. I recommend you check out the official documentation to see what other cool stuff is possible.

# data[].attributes.attachment_link.title
What can you do with PostgreSQL and JSON?

# data[].attributes.attachment_link.description
PostgreSQL 9.2 added a native JSON data type, but didn't add much else. You've got three options if you actually want to do something with it: Wait for PostgreSQL 9.3 (or use the beta) Use the plv8 extension.

# data[].attributes.headline
PostgreSQL: Documentation: 9.1: GiST and GIN Index Types

# data[].attributes.subject
PostgreSQL: Documentation: 9.1: GiST and GIN Index Types

# data[].attributes.attachment_link.title
PostgreSQL: Documentation: 9.1: GiST and GIN Index Types

# data[].attributes.attachment_link.description
Lossiness causes performance degradation due to unnecessary fetches of table records that turn out to be false matches. Since random access to table records is slow, this limits the usefulness of GiST indexes. The likelihood of false matches depends on several factors, in particular the number of unique words, so using dictionaries to reduce this number is recommended.

# data[].attributes.headline
Evaluation Criteria for Schemaless database

# data[].attributes.subject
Evaluation Criteria for Schemaless database

# data[].attributes.body
Look at Mongo! Just look at it! This was from 2013.

I think we need to move away from MongoDB.

# data[].attributes.headline
hstore vs. JSON - Which to use in Postgres Craig Kerstiens

# data[].attributes.subject
hstore vs. JSON - Which to use in Postgres Craig Kerstiens

# data[].attributes.attachment_link.title
hstore vs. JSON - Which to use in Postgres - Craig Kerstiens

# data[].attributes.attachment_link.description
If you're deciding what to put in Postgres and what not to, consider that Postgres can be a perfectly good schema-less database. Of course as ...

# data[].attributes.headline
KLOTH.NET - DIG - DNS lookup - find IP address

# data[].attributes.subject
KLOTH.NET - DIG - DNS lookup - find IP address

# data[].attributes.attachment_link.keywords[]
dig dns look up find ip address info information dns cname ptr soa mx txt ns
   domain name system query addresses resolve reverse lookup own owner whois
   look-up record computer get server status internet protocol networking nslookup
   convert translate host name ip address conversion translation books reading

# data[].attributes.attachment_link.title
KLOTH.NET - DIG - DNS lookup - find IP address

# data[].attributes.attachment_link.description
DIG - use this online web tool to query a DNS nameserver to look up and find IP address information of computers in the internet

# data[].attributes.headline
ddos - How can I identify open DNS resolvers in my network? - Information Security Stack Exchange

# data[].attributes.subject
ddos - How can I identify open DNS resolvers in my network? - Information Security Stack Exchange

# data[].attributes.attachment_link.readable_text
Identify an Open DNS server by your own querying via NMAP:

x.x.x.x = DNS server IP
nmap -sU -p 53 -sV -P0 --script "dns-recursion" x.x.x.x
Possible output would be:

PORT STATE SERVICE VERSION
53/udp open domain ISC BIND "version"
*|_dns-recursion: Recursion appears to be enabled*

If you prefer make use of online services, openresolver project is very good, it checks also subnets of /22 width, so check it out ---> http://www.openresolverproject.org

After an Open DNS server discovery with online tools is a good idea to do a double check getting a proof about recursion ---> http://www.kloth.net/services/dig.php

Example output, watch the "ra" flag means recursion available:
; <<>> DiG 9.7.3 <<>> @x.x.x.x domain.cn A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: xxx
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 5, ADDITIONAL: 0

DISABLING RECURSION
( source knowledgelayer softlayer com)

Disable Recursion in Windows Server 2003 and 2008

Access the DNS Manager from the Start menu: Click the Start button. Select Administrative Tools. Select DNS. Right click on the desired DNS Server in the Console Tree. Select the Proprerties tab. Click the Advanced button in the Server Options section. Select the Disable Recursion checkbox. Click the OK button.

Disable Recursion in Linux

Locate the BIND configuration file within the operating system. The BIND configuration file is usually located in one of the following paths: /etc/bind/named.conf /etc/named.conf Open the named.conf file in your preferred editor. Add the following details to the Options section: allow-transfer {"none";}; allow-recursion {"none";}; recursion no; Restart the device.

# data[].attributes.attachment_link.title
How can I identify open DNS resolvers in my network?

# data[].attributes.attachment_link.description
Identify an Open DNS server by your own querying via NMAP: x.x.x.x = DNS server IP nmap -sU -p 53 -sV -P0 --script "dns-recursion" x.x.x.x Possible output would be: PORT STATE SERVICE VERSION 53/udp open domain ISC BIND "version" *|_dns-recursion: Recursion appears to be enabled* If you prefer make use

# data[].attributes.attachment_link.description
Open Resolvers pose a significant threat to the global network infrastructure by answering recursive queries for hosts outside of its domain. They are utilized in DNS Amplification attacks and pose a similar threat as those from Smurf attacks commonly seen in the late 1990s.

# data[].attributes.body
Using the GIN (Generalized Inverted index) vs GIST (Generalized search tree) index. Differences are highlighted here.

# data[].attributes.attachment_link.description
This module implements the hstore data type for storing sets of key/value pairs within a single PostgreSQL value. This can be useful in various scenarios, such as rows with many attributes that are rarely examined, or semi-structured data. Keys and values are simply text strings.

# data[].attributes.body
Here's something interesting: when MongoDB trashes (as in previous post), the response time also increases.

# data[].attributes.body
Just got a notice from New Relic that MongoDB (or pioneer) had disk IO > 90%. Suspect Mongo flushing cached data to disk.

# data[].attributes.body
Indexes should be built according to how data is being queried.

{  "t": <time>,

   "d": {
     "id": <object_id>,
     "resource": "walls",
     "action": "update",
....

   }

}
The query for the activity feed for a user as of now is two OR clauses. They are:

1. { 'd.builder_id': <curr_user_id>, 'd.resource': 'walls', 'd.action': 'update', 'd.user_id': { '$ne': <curr_user_id>' }}

2. {'d.wall_id': { '$in': [<wall_1>,<wall_2>...]}, 'd.resource': 'wishes', 'd.user_id': {'$ne': <curr_user_id>}}

We then sort the results by time.

I built the following indexes.

['d.builder_id', 'd.resource', 'd.action', 'd.user_id', 't']

['d.wall_id', 'd.resourc'e, 'd.user_id', 't']

The first index works for the first OR query. But the second, not so much.

This is because each mongo query can only use 1 index.

The next is because the 2nd clause in the OR query uses $in, thus it cannot reference a contiguous series of entries in an index, hence Mongo cannot use it.

The other way is to run (say) 100 mongo queries, 1 for each wall that should have been in the $in query, and merge them based on time (using the merge operation in a merge sort). This is too painful.

We can also change the query to not use $in and just fetch all wish events for all walls, then filter document by document. But that's probably the slowest way to do it, based on the profiling.

The other alternative is to write to a separate collection. Or write to a separate postgres instance. Or use elasticsearch.

Going to do some reading up on elasticsearch.

# data[].attributes.body
written in Jan 2014 on Quora.

Summary:

- This link isn't really for ElasticSearch as a data store

- ElasticSearch is distributed. Its benefits are realised when it is used that way.

- ElasticSearch requires more than half the node count as master nodes to avoid split brain issue.

- if you have great deal of data but limited resources, ElasticSearch might not be good.

- Under the same question but in a different answer, a user talks about how big a memory hog ElasticSearch is.

# data[].attributes.attachment_link.readable_text
Basically, Elastic is wonderful so long as you do everything right and stay well within the bounds of its intended usage. It merely doesn't fail gracefully, in particular situations. (In other situations, it fails quite well: on a multi-node cluster, shards will be replicated if any node goes down.)

Other, less extreme reasons to avoid Elastic:

1) The documentation is written very much in a tutorial style, intended for people using it in exactly the way the developers intend it to be used. In theory, lots of obscure uses are supported -- it's just that documentation for these use cases is either nonexistent or hidden in the middle of documentation for a very dissimilar use case. If you intend to do nothing but index some data and then retrieve it later using very straightforward queries, you will have no trouble; if you intend to take pages of fairly complex SQL and turn them into ES queries, you'll thirst for a plain old reference manual with a list of keywords and some BNF.

2) The default settings are stupid for anything other than a toy install, but they will work misleadingly well until you run into your first major problem. For instance, there's a default limit for the size of the buffer used in handling POST data that's something like 100mb -- completely unreasonable for bulk loading, but something that works just fine on tiny test loads. If you have a multi-node cluster, you will need to ignore the comment that tells you that minimum master nodes = 3 is reasonable for any system with greater than one node, and instead use just over half the node count or else you'll get a split brain issue 100% of the time. Nearly all the documentation is about the simple load mechanism, but you will need to use the bulk load mechanism whenever you have a lot of data to load (for instance, when porting over data previously stored in some other DB) because simple loads are slow -- but bulk loads have all sorts of annoying restrictions (for instance, you must use contiguous document IDs).

3) If you don't have a multi-node cluster, you probably won't benefit much at all from it. Nearly all of Elastic's speed compared to Lucene comes from pure parallelism (and the splitting of indexes into many shards); a single-shard single-machine elastic install will be almost identical in performance to a Lucene install. A 3-node cluster with a shard for every hardware core, on the other hand, will be a constant factor c=nodecount*cpucount faster on the worst searches and even better on the easier ones. Keep in mind that you basically have linear scaling by total shard count for the entire cluster (not counting replica shards). If you're going to take your one single-core database machine and dump an elastic install on it, you might as well not bother.

If, on the other hand, you have 500 machines with half a TB of ram each, all in the same data center, and you're willing to dedicate them all to simple lookups, you're golden (and despite the exaggeration, this general situation is not uncommon amongst high-profile Elastic users; foursquare uses Elastic this way from what I can discern, grabbing whole groups of fields based on fairly straightforward geolocation searches).

# data[].attributes.attachment_link.title
Why shouldn't I use ElasticSearch as my primary datastore?

# data[].attributes.attachment_link.description
John Ohno's answer: If you have a great deal of data and limited resources, ElasticSearch (while probably usable) will be a headache to get working. It's very easy to exhaust resources, and Elastic has absolutely no safeguards or sensible fallbacks in the case of overrunning either JVM-allocated ...

# data[].attributes.body
Really simple tutorian on HStore. But good to know.

# data[].attributes.attachment_link.readable_text
This post covers Elasticsearch 1.1.0. In the months since its publication, Elasticsearch has added a comprehensive overview of correctness issues and their progress towards fixing some of these bugs.
Previously, on Jepsen, we saw RabbitMQ throw away a staggering volume of data. In this post, we'll explore Elasticsearch's behavior under various types of network failure.

Elasticsearch is a distributed search engine, built around Apache Lucene-a well-respected Java indexing library. Lucene handles the on-disk storage, indexing, and searching of documents, while ElasticSearch handles document updates, the API, and distribution. Documents are written to collections as free-form JSON; schemas can be overlaid onto collections to specify particular indexing strategies.

As with many distributed systems, Elasticsearch scales in two axes: sharding and replication. The document space is sharded-sliced up-into many disjoint chunks, and each chunk allocated to different nodes. Adding more nodes allows Elasticsearch to store a document space larger than any single node could handle, and offers quasilinear increases in throughput and capacity with additional nodes. For fault-tolerance, each shard is replicated to multiple nodes. If one node fails or becomes unavailable, another can take over. There are additional distinctions between nodes which can process writes, and those which are read-only copies-termed "data nodes"-but this is primarily a performance optimization.

Because index construction is a somewhat expensive process, Elasticsearch provides a faster, more strongly consistent database backed by a write-ahead log. Document creation, reads, updates, and deletes talk directly to this strongly-consistent database, which is asynchronously indexed into Lucene. Search queries lag behind the "true" state of Elasticsearch records, but should eventually catch up. One can force a flush of the transaction log to the index, ensuring changes written before the flush are made visible.

But this is Jepsen, where nothing works the way it's supposed to. Let's give this system's core assumptions a good shake and see what falls out!

What does the docs say?

What is the Elasticsearch consistency model? When I evaluate a new database, I check the documentation first.

The docs tell us that Elasticsearch provides optimistic concurrency control: each document has an atomic version number, and updates can specify a particular version required for the write to go through; this allows atomic CAS and provides the basis for independently linearizable updates to each document; i.e. every update to a particular document appears to take place atomically between the start of a request and its completion.

Moreover, Elasticsearch requires synchronous acknowledgement of each write from a majority of nodes, which suggests that a.) nodes in the minority side of a network partition will reject writes, and b.) acknowledged writes will be durable in the event that a new primary node is elected. From the documentation:

To prevent writes from taking place on the "wrong" side of a network partition, by default, index operations only succeed if a quorum (>replicas/2+1) of active shards are available.

By default, the index operation only returns after all shards within the replication group have indexed the document (sync replication).

Synchronous replication to a majority of replicas, plus the use of version-based CAS, suggests that Elasticsearch has all the primitives required to treat documents as linearizable registers. Moreover, if we issue a flush command, Elasticsearch will apply any outstanding writes from the transaction log to Lucene; a composite flush+ search operation should also be linearizable, much like Zookeeper's sync+ read.

So: the consistency docs paint an optimistic picture-but what does Elasticsearch do when a node or network fails?

A search for partition on elasticsearch.org site reveals only a single reference-the "on the 'wrong' side of a network partition" comment regarding quorum writes. A search for fault tolerance returns only a single result- an overview blog post which simply describes Elasticsearch as "resilient to failing nodes".

The results for "failure" are a bit more comprehensive, though. Coping with failure, part of the introductory tutorial, explains that after killing a node,

The first thing that the new master node did was to promote the replicas of these shards on Node 2 and Node 3 to be primaries, putting us back into cluster health yellow. This promotion process was instantaneous, like the flick of a switch.

I like "instantaneous" promotion. Fast convergence times are key for testing a system in Jepsen; systems that take minutes to converge on a new state drag out the testing process, especially when testing eventual consistency.

So why is our cluster health yellow and not green? We have all 3 primary shards, but we specified that we wanted two replicas of each primary and currently only one replica is assigned. This prevents us from reaching green, but we're not too worried here: were we to kill Node 2 as well, our application could still keep running without data loss because Node 3 contains a copy of every shard.

So Elasticsearch claims to cope with the loss of a majority of nodes. That could be a problem from a consistency standpoint: it means that reads might, depending on the implementation, be able to read stale data. As we saw in the RabbitMQ post, stale reads rule out a number of consistency models. We'll be careful to wait for the cluster to report green before doing a read, just in case.

The broader landscape

So the official documentation is inspecific about fault tolerance. The next thing I look at in evaluating a database is reports from users: Blog posts, mailing list questions, and so on. Let's start with a search for Elasticsearch partition tolerance.

Result #1 is this mailing list discussion from 2010, where Tal Salmona asks what ElasticSearch guarantees with respect to the CAP theorem. Shay Banon, the primary author of Elasticsearch, responds:

When it comes to CAP, in a very high level, elasticsearch gives up on partition tolerance. This is for several reasons:

2010 was a simpler time.

I personally believe that *within the same data center", network partitions very rarely happen, and when they do, its a small set (many times single) machine that gets "partitioned out of the network". When a single machine gets disconnected from the network, then thats not going to affect elasticsearch.

Quantitative evidence is hard to find, but we have some data. Rare events are still worth defending against. Still, this is somewhat reassuring: single-node partitions were a design consideration, and we can expect Elasticsearch to be robust against them.

When it comes to search engines, and inverted index, its very hard to the point of impossible to build a solution that tries to resolve consistency problems on the fly as most products do (the famous "read repair"). When you search, you search over a large amount of docs and you can't read repair each one all the time. Key value / column based solutions have life easy

This is true! Search is a hard problem to distribute. It doesn't tell us much about Elasticsearch's behavior, though. Sergio Bossa asks what happens in a simple partitioned scenario, and Banon responds with some very interesting discussion of the general problem, but not many specifics. One key point:

One simple option to solve this is to have the user define a "minimum" size of cluster that should be up, and if its not, the nodes will not allow writes. This will be in elasticsearch and its actually not that difficult to implement.

This is now built into Elasticsearch as an option called discovery.zen.minimum_master_nodes. As numerous users have noted, Elasticsearch will happily allow concurrent primary nodes when this number is less than n/2+1. Why is it even legal to set it lower? I'm not sure, but we'll make sure to set the minimum_master_nodes to 3 for our five-node cluster.

All this is old news, though. The next interesting result is a mailing list discussion from early 2014, in which Nic asks for a clarification of Elasticsearch's CAP tradeoffs, given the 2010 thread.

From my understanding it seems like Kimchy was confused here. As a distributed system ES can't give up on the P - you can't will network/communication failures out of existent!

Instead, it seems like ES mostly compromises on the A (availability) part of CAP. For example, unless you are willing to suffer potential split-brain scenarios, setting min master nodes to n/2 + 1 will mean the smaller group under a network partition will become unavailable (it will not respond to read/writes). If you do allow split-brain then clearly consistency is compromised and the client service will need to have some kind of conflict resolution mechanism.

My thoughts exactly. What's going on here?

It would be great if there were a page on the ES site/guide which went into these issues in more detail as it is (IMO) essential information in understanding how ES works and in deciding whether it is appropriate for your use case.

Yes, yes, a thousand times yes. One of the principle reasons I put so much time into Jepsen is to prompt database vendors to clearly explain their safety and liveness invariants.

Jörg Prante, a regular on the mailing list, responds:

ES gives up on partition tolerance, it means, if enough nodes fail, cluster state turns red and ES does not proceed to operate on that index.

That sounds like giving up availability to me. Remember, CAP A means that every request to a non-failing node completes successfully.

ES is not giving up on availability. Every request will be responded, either true (with result) or false (error). In a system being not available, you would have to expect the property of having some requests that can no longer be answered at all (they hang forever or the responder is gone).

Errors are not successful responses; they don't count towards availability.

The principle design of distributed operations in ES is like this: write all ops on an index into a WAL (the translog). Send the ops to the nodes while even some nodes may work reliable, some not. Stopping a node does not harm the cluster as long as the replica level is high enough. When a stopped node rejoins, initiate a recovery, using the WAL. Let the "best" WAL result of all consistent results of the replica win for recovering the index state.

This is useful, concrete information, and appears to describe something like a CP system, but given the poster's confusion with respect to CAP I'd like a more authoritative explanation.

Itamar Syn-Hershko responds to Prante, arguing Elasticsearch gives up consistency when configured with fewer than quorum primary-eligible nodes. This seems valid to me, but I'm curious what happens when we do use the quorum constraints correctly.

The very fact that you can have enough replicas of your index which will make the cluster never get to a red state (e.g. the number of nodes you have) proves this. An index (an eventually, all indexes on your cluster) can survive a network split. It can also be always available, hence ES is AP.

Elasticsearch's compromise is on C - consistency - like most NoSQL databases. It uses Eventual Consistency to answer queries, not just because of NRT search, but also because you may be querying a replica (a slave node) which hasn't been brought up to speed yet.

Writes may be linearizable even if reads aren't. We might be able to get Elasticsearch to do CAS safely, and test it by waiting for the cluster to recover and performing an index flush before read.

Prante responds, asserting that Elasticsearch compare-and-set operations are linearizable.

Consistency is not given up by ES. First, on doc level, you have "write your own read" consistency implemented as versioning - a doc read followed by a doc write is guaranteed to be consistent if both read and write versions match (MVCC).

Replica are not interfering with consistency, they are for availability.

As we've discussed, replicas are the reason why consistency is hard in the first place.

split brains can happen and ES can happily proceed reading and writing to the index in such a case, but the result is not predictable - the usual case is that two masters are going to control two divergent indices, and that is catastrophic. This is not a fault but a (nasty) feature, and must be controlled by extra safeguarding, by setting the minimum master value in the configuration ...

Well, I'm a little confused by all this, but the impression that I get is that if we set minimum_master_nodes to a majority of the nodes in the cluster, Elasticsearch should be safe.

That's really it for partition discussion, as far as I can tell. No official docs explaining the resolution algorithm, and some possibly outdated, contradictory discussion on a mailing list. There are a number of blog posts that repeat these sources, and some meetup discussions, but nothing concrete.

Nontransitive partitions

Then somebody pointed me at this GitHub issue, reported by Saj Goonatilleke in December 2012. It's titled "minimum_master_nodes does not prevent split-brain if splits are intersecting." That sounds interesting.

With this setup, I can easily split a 3-node cluster into two 'hemispheres' (continuing with the brain metaphor) with one node acting as a participant in both hemispheres. I believe this to be a significant problem, because now minimum_master_nodes is incapable of preventing certain split-brain scenarios.

In Saj's test case, a split in a three-node cluster isolates two nodes from each other, but not from a common node visible to both. Because both isolated nodes can see two thirds of the cluster (themselves and the common node), they believe they are eligible for leader election even when minimum_master_nodes is at least a majority.

This points to a deeper problem in Elasticsearch's leader election process. Most leader election algorithms divide time into monotonic epochs or terms, and allow only one node to become the leader for a given term. They often enforce this constraint by requiring that a given node can only support one candidate node per term. This property, combined with the constraint that a candidate must receive votes from a majority to become leader for a term, ensures that there will be at most one leader per term.

ZenDisco, Elasticsearch's cluster membership system, has no such invariant. A node will happily support two leaders simultaneously.

Some users reported success in switching to the Zookeeper plugin for leader election, but that component has been, as far as I know, broken since 0.90.6.

Building a nemesis

This sort of scenario isn't hard to verify with Jepsen, and it's a good opportunity to introduce a Jepsen concept that we haven't explored yet: the nemesis. Nemeses are a special type of Jepsen client, but instead of performing operations against a node in the database, they play havoc with the entire cluster; severing network links, adjusting clocks, killing processes, causing single-bit errors on disk, and so on.

First things first: we'll use iptables to drop traffic from a node, or a collection of nodes.

Next, we need a way to fully describe a given network topology. One way would be as an adjacency list, but for our purposes it's a little clearer to describe the links that we're going to cut.

What kind of grudge might we want to hold? Let's call a complete grudge one in which all links are symmetric: if A can talk to B, B can talk to A, and vice-versa. We'll be nice to our networks, and avoid unidirectional partitions for now-though they can happen in production, and Magnus Haug reported just last week that asymmetric partitions will wedge an Elasticsearch cluster in a split-brain state.

We want to cut the network roughly in half, so we'll take a list of nodes and chop it into two pieces. Then we can feed those pieces to complete-grudge to compute the links we need to cut to isolate both nodes into their own network components.

Finally, a slight modification to the complete grudge: we're going to allow one node to talk to everyone.

Last piece of the puzzle: we've got to respond to operations by initiating and healing the network partition we've computed. This partitioner nemesis is generic-it takes a pluggable function, like bridge, to figure out what links to cut, then calls partition! with those links.

Let's give it a shot. In this test we'll implement a linearizable set which supports adding elements and reading the current set, just like we did with Etcd, by using [Elasticsearch's compare-and-set primitives]( https://github.com/aphyr/jepsen/blob/1403f2d6e61c595bafede0d404fd4a893371c036/elasticsearch/src/jepsen/system/elasticsearch.clj#L161-L226. Then we'll write a register test that enqueues integers (slightly staggered in time to avoid too many CAS failures) , while cutting the network into the nontransitive partition shape reported in the ticket.

Speed bumps

The first thing I found was, well, nothing. The test crashed with a cryptic error message about exceptions-and it took a while to figure out why.

Elasticsearch supports two protocols: an HTTP protocol, which is slower, and a native binary protocol. In most cases you want to use the native protocol where available. Unfortunately, Elasticsearch uses Java serialization for exceptions-which is problematic because different JVMs serialize classes differently. In particular, InetAddress's representation is not stable across different JVM patchlevels, which means that servers or even clients running different JVM versions will explode when trying to parse an error message from another node.

Another roadblock: "instantaneous" cluster convergence isn't. It takes Elasticsearch ninety seconds (three rounds of 30-second timeouts) to detect a node has failed and elect a new primary. We'll have to slow down the test schedule, allowing the network to stabilize for 300 seconds, in order to get reliable cluster transitions. There's a configuration option to adjust those timeouts...

# Set the time to wait for ping responses from other nodes when discovering. # Set this option to a higher value on a slow or congested network # to minimize discovery failures: # discovery.zen.ping.timeout: 3s

But as far as I can tell, that knob doesn't make failure detection any faster.

[2014-06-16 17:25:40,901][INFO ][discovery.zen ] [n5] master_left [[n1][9fvd-dZBTo2re4dukDuuRw][n1][inet[/192.168.122.11:9300]]], reason [failed to ping, tried [3] times, each with maximum [30s] timeout]

Elasticsearch has some really terrific tools for cluster introspection: you can request a JSON dump of the cluster status easily via CURL, which makes writing tests a breeze. You can also-and this is really awesome-block until the cluster reaches some desired status. In the Jepsen client, I block until the cluster health reaches green before starting the test, and once the network is repaired, before doing a final read of the set.

The problem is that the health endpoint will lie. It's happy to report a green cluster during split-brain scenarios. I introduced additional delays to the test schedule to give Elasticsearch even more time to recover, with mixed success. Sometimes the cluster will wedge hard, and refuse to make progress until I start bouncing nodes.

Anyway, with those caveats: results!

Nontransitive partition results

Elasticsearch compare-and-set isn't even close to linearizable. During this type of network partition, both primaries will happily accept writes, and the node in the middle... well, it's caught in a three-way call with Regina George, so to speak. When the cluster comes back together, one primary blithely overwrites the other's state.

Knossos reported a linearization failure-but the final state was misssing tons of writes, and Knossos panics as soon as a single one is lost. "Welp, my job is done here: peace".

But we're not just interested in whether the system is linearizable. We want to quantify just how much data might be corrupted. I wrote a custom checker for Elasticsearch to try and quantify that write loss, similar to the RabbitMQ queue checker.

Here's a result from a typical run with that checker

645 out of 1961 writes acknowledged then lost. Only 1103 writes made it to the final read. Elasticsearch issued more false successes than failure results.

Clearly Elasticsearch's MVCC support is neither consistent nor available; it happily allows two concurrent primaries to accept writes, and destroys the writes on one of those primaries when the partition resolves, but the majority-write constraint prevents Elasticsearch from offering total availability.

What's really surprising about this problem is that it's gone unaddressed for so long. The original issue was reported in July 2012; almost two full years ago. There's no discussion on the website, nothing in the documentation, and users going through Elasticsearch training have told me these problems weren't mentioned in their classes. Fixing a distributed system is hard, but documenting its gotchas is easy. Look to RabbitMQ or Riak, both of which have extensive documentation around failure recovery, and the scenarios in which their respective systems could lose or corrupt data.

This is not a theoretical constraint. There are dozens of users reporting data loss in this ticket's discussion, and more on the mailing list. Paul Smith reports

I'm pretty sure we're suffering from this in certain situations, and I don't think that it's limited to unicast discovery.

We've had some bad networking, some Virtual Machine stalls (result of SAN issues, or VMWare doing weird stuff), or even heavy GC activity can cause enough pauses for aspects of the split brain to occur.

Ivan Brusic confirms:

I have seen the issue on two different 0.20RC1 clusters. One having eight nodes, the other with four.

Trevor Reeves observes:

We have been frequently experiencing this 'mix brain' issue in several of our clusters - up to 3 or 4 times a week. We have always had dedicated master eligible nodes (i.e. master=true, data=false), correctly configured minimum_master_nodes and have recently moved to 0.90.3, and seen no improvement in the situation.

From Elad Amit:

We just ran into this problem on a 41 data node and 5 master node cluster running 0.90.9

And Mark Tinsley:

I have been having some strange occurrences using elasticsearch on aws.... For some reason, node A and B cannot talk to each other... but both can still talk to C and C can talk to A and B i.e. a 'on the fence' network partition as C can still see all.... As you can see B is now a new master but A has not been removed as a master, because A can still see C so has the minimum master node criteria satisfied.

Some of these problems can be ameliorated by improving Elasticsearch's GC performance and overall responsiveness-but this will only reduce the frequency of split-brain events, not fix the underlying problem. Elasticsearch needs a real consensus algorithm to linearize updates to documents.

Needless data loss

So let's back off. Maybe version control is a lost cause-but there is something Elasticsearch can definitely handle safely, and that's inserts. Inserts will never conflict with one another, and can always be merged with set union. Updates and deletes to those documents may not linearize correctly, but we can bias towards preserving data. Moreover, insertions can theoretically be 100% available: every node can take writes and queue them for later, modulo durability constraints.

Here's a client which implements a set of integers by inserting a fresh document with an auto-generated ID for each element.

The results? Massive write loss.
Of two thousand attempts, six hundred eighty eight documents were acknowledged by Elasticsearch then thrown away.
This is something the ticket didn't make clear: Elasticsearch could merge divergent replicas by preserving inserts on both nodes-but it doesn't. Instead it throws all that data away. In fact, sometimes you'll get mind-boggling errors like

{ :body { "error" : "RemoteTransportException[ [Cadaver][inet[/192.168.122.12:9300]][index]]; nested: RemoteTransportException[ [Death Adder][inet[/192.168.122.11:9300]][index]]; nested: DocumentAlreadyExistsException[ [jepsen-index][1] [number][EpVU56YERBOfRqyVc-_hAg]: document already exists]; ", "status" : 409 }}

Elasticsearch claims-and Wireshark traces confirm-that documents inserted without an ID will receive an auto-generated UUID for their document ID. How is it possible that an insert of a fresh document with a fresh UUID can fail because that document already exists? Something is seriously wrong here.

Random transitive partitions

Maybe you don't believe that nontransitive-e.g. bridged-network partitions are all that common. Can Elasticsearch handle a complete, disjoint partition which isolates the cluster cleanly into two halves, without a node connecting the two sides?

Here's a nemesis to generate this kind of partition. To determine the grudge-the links to cut-we'll shuffle the list of nodes randomly, cut it in half, and isolate both halves from each other completely. We'll use comp again to compose several functions together.

Because this partitioner shuffles the nodes, each time we invoke it we'll get a different minority and majority component to the partition. This is a pattern that a CP system can handle well: because a majority is always visible, after a brief reconvergence latency spike a linearizable database could offer availability throughout this test.

What does Elasticsearch do in this scenario?

Elasticsearch loses data even when the partitions are total. You don't need a bridge node to make replicas diverge. Again, we're not even talking CAS updates-these are all freshly inserted documents; no changes, no deletions. Every single one of these writes could have been preserved safely.

Only 25 writes thrown away this round, but that number should have been zero. You can lower the probability of loss by adding more replicas, but even with a replica on every single node, Elasticsearch still finds a way to lose data.

Fixed transitive partitions

Maybe the problem is that the partitions are shifting over time-perhaps isolating one set of nodes, then another set, is what confuses Elasticsearch. We might be looking at a time horizon too short for Elasticsearch to stabilize between failures, causing it to interpret the disjoint components as some partially-transitive graph.

But as Shikhar Bhushan noted last week, even a nemesis which always generates the same partitioning pattern, like [n1 n2] [n3 n4 n5], causes split brain and write loss. This nemesis is just like the random partitoner, but we drop the shuffle step.

Running this test reveals that it's not just shifting partitions: a constant partition pattern will cause Elasticsearch to enter split brain and throw away data.

Seventeen writes lost here. It doesn't take a shifting pattern of failures to break the cluster.

Single-node partitions

Maybe this is to be expected; Banon asserted on the mailing list that partitions involving multiple nodes are rare-but Elasticsearch should be able to handle a single node being isolated.

By symmetry, the most interesting node to isolate is the primary. To find the primary, we can query Elasticsearch's cluster status endpoint on each host, to see what it thinks the current primary is.

We'll take the nodes, make an HTTP request to each, and extract the master_node field of the result. That's a unique instance identifier, so we dig into the nodes structure to map that identifier back to a node name. Then we turn those [node, supposed-primary] pairs into a hashmap.

Let's limit ourselves to nodes that think they're the primary.

We're just filtering that map to pairs where the key and value are equal, then returning the keys-nodes that think they themselves are current primaries. Next, we'll write a special partitioner, just for Elasticsearch, that isolates those nodes.

The first time we run this, a single node will be isolated. We'll let the cluster converge for... say 200 seconds-more than long enough for pings to time out and for a new primary to be elected, before repairing the partition. Then we'll initiate a second partition-again cutting off every node that thinks it's a primary. There might be more than one if Elasticsearch is still in split-brain after those 200 seconds.

Elasticsearch still loses data, even when partitions isolate only single nodes.

This test lost eight acknowledged writes, all clustered around write number 630, just prior to the end of the second partition. It also wedges the cluster hard-even though cluster status reports green, some nodes refuse to converge. All kinds of weird messages in the logs, including an enigmatic reason [do not exists on master, act as master failure]. In order to get any data back from the final read in these tests, I had to manually restart the stuck nodes during the end-of-test waiting period. Anecdotal reports suggest that this is not merely a theoretical problem; Elasticsearch convergence can deadlock in production environments as well.

One single-node partition

Perhaps isolating nodes more than once is too difficult to handle. Let's make things easier by only having one partition occur. We'll cut off a single primary node once, wait for the cluster to converge, let it heal, wait for convergence again, and do a read. This is the simplest kind of network failure I know how to simulate.

This time the cluster manages to recover on its own, but it still lost inserted documents.

31 acknowledged writes lost, out of 855 attempts. Or, if you're unlucky, it might throw away almost all your data.

In this run, a single network partition isolating a primary node caused the loss of over 90% of acknowledged writes. Of 619 documents inserted, 538 returned successful, but only 54-10%-of those documents appeared in the final read. The other 484 were silently discarded.

Remember, this is the one kind of network failure Elasticsearch was designed to withstand. Note that the write loss pattern varies with time; we lost a mixture of writes through the first quarter of the test, and recovered nothing written after 133.

This test is a little harder to reproduce; sometimes it'll recover all writes, and occasionally it'll drop a ton of data. Other times it just loses a single write. I thought this might be an issue with flush not actually flushing the transaction log, so I upped the delay before read to 200 seconds-and still found lost data. There may be multiple bugs at play here.

Recommendations for Elasticsearch

To summarize: Elasticsearch appears to lose writes-both updates and even non-conflicting inserts-during asymmetric partitions, symmetric partitions, overlapping partitions, disjoint partitions, and even partitions which only isolate a single node once. Its convergence times are slow and the cluster can repeatably deadlock, forcing an administrator to intervene before recovery.

I wish I had a better explanation for these problems, but I've already burned a hundred-odd hours on Elasticsearch and need to move on to other projects. I've got a few conjectures, though.

Elasticsearch's cluster membership protocol is a mess, and Elasticsearch opts against using an external coordination service like Zookeeper for membership on the grounds that it has access to less information about the cluster's ongoing operations. Elasticsearch has started to put work into their fault-tolerance mechanisms, but that work is still incomplete.

The thing is that leader election is a well-studied problem with literally dozens of formally described algorithms and none of that literature appears to have influenced Zendisco's design. Leader election is not sufficient to ensure correctness, but using a peer-reviewed algorithm would be a heck of a lot better than the current morass of deadlocks and persistent split-brain.

Despite what the docs claim, I'm not convinced that Elasticsearch actually does synchronous replication correctly. It looks to me-and I haven't verified this with a packet trace yet, so this is a bit shaky-that primary nodes can acknowledge writes made even when they're isolated from the rest of the cluster. The patterns of write loss in the Jepsen histories are pretty darn suggestive. I'd expect to see a clear-cut pattern of timeouts and failures when a partition starts, and no successful writes until a new primary is elected. Instead, Elasticsearch continues to acknowledge writes just after a partition occurs. It might be worth investigating that write path to make sure it won't ack a write until a majority confirm.

Speaking of majority, Shikhar Bhushan notes that Elasticsearch considers a quorum for two nodes to be a single node. This definition invalidates, well, basically every proof about non-dominating coterie consensus, but the Elasticsearch team doesn't seem keen on changing it. I don't think this affects the tests in Jepsen, which maintain three or five replicas, but it's worth keeping in mind.

None of these changes would be sufficient for linearizability, incidentally. Distributed commit requires a real consensus protocol like Paxos, ZAB, Viewstamped Replication, Raft, etc. The Elasticsearch team is investigating these options; from unofficial murmurings, they may release something soon.

In the short term, Elasticsearch could provide better safety constraints by treating a collection of documents as a CRDT -specifically, an LWW-element set where the timestamps are document versions. This isn't linearizable, but it would allow Elasticsearch to trivially recover inserts made on both sides of a cluster, and to preserve writes made on only one side of a partition. Both of these behaviors would significantly improve Elasticsearch's fault tolerance, and is significantly simpler to implement than, say, Raft or Paxos.

Recommendations for users

If you are an Elasticsearch user (as I am): good luck.

Some people actually advocate using Elasticsearch as a primary data store; I think this is somewhat less than advisable at present. If you can, store your data in a safer database, and feed it into Elasticsearch gradually. Have processes in place that continually traverse the system of record, so you can recover from ES data loss automatically.

Folks will tell you to set minimum_master_nodes to a majority of the cluster, and this is a good idea-but it won't, by any means, buy you safety. I'm honestly not sure what will. Raising the replica count reduces the odds of losing inserts, but means more replicas to conflict during CAS and won't guarantee safety even when a replica is present on every single node in the cluster.

The good news is that Elasticsearch is a search engine, and you can often afford the loss of search results for a while. Consider tolerating data loss; Elasticsearch may still be the best fit for your problem.

Moreover, none of this invalidates the excellent tooling available around Elasticsearch. People love Logstash and Kibana. The ES API is surprisingly straightforward, and being able to just spew JSON at a search system is great. Elasticsearch does a great job of making distributed search user-friendly. With a little work on correctness, they can make it safe, too.

This concludes the third round of Jepsen tests. I'd like to thank Comcast for funding this research, and to Shay Banon, Bob Poekert, Shikhar Bhushan, and Aaron France for their help in writing these tests and understanding Elasticsearch's behavior. I am indebted to Camille Fournier, Cliff Moon, Coda Hale, JD Maturen, Ryan Zezeski, Jared Morrow, Kelly Sommers, Blake Mizerany, and Jeff Hodges for their feedback on both the conference talks and these posts, and to everyone who reached out with reports of production failures, paper recommendations, and so much more.

And finally, thank you, reader, for sticking with what was probably far more than you ever wanted to know about databases and consistency. I hope you can use the techniques from these posts to help verify the safety of your own distributed systems. Happy hunting.

# data[].attributes.attachment_link.description
This post covers Elasticsearch 1.1.0. In the months since its publication, Elasticsearch has added a comprehensive overview of correctness issues and their progress towards fixing some of these bugs. Previously, on Jepsen, we saw RabbitMQ throw away a staggering volume of data. In this post, we'll explore Elasticsearch's behavior under various types of network failure.