# data[].attributes.attachment_link.readable_text
Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Asked

I am an IT student and I am now in the 3rd year in university. Until now we've been studing a lot of subjects related to computers in general (programming, algorithms, computer architecture, maths, etc).

I am very sure that nobody can learn every thing about security but sure there is a "minimum" knowledge every programmer or IT student should know about it and my question is what is this minimum knowledge?

Can you suggest some e-books or courses or anything can help to start with this road?

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.

Stack Overflow works best with JavaScript enabled

# data[].attributes.attachment_link.title
What should every programmer know about security?

# data[].attributes.attachment_link.description
Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now Asked I am an IT student and I am now in the 3rd year in university. Until now we've been studing a lot of subjects related to computers in general (programming, algorithms, computer architecture, maths, etc).

# data[].attributes.attachment_link.readable_text
The idea here is that most of us should already know most of what is on this list. But there just might be one or two items you haven't really looked into before, don't fully understand, or maybe never even heard of.
Interface and User Experience

Be aware that browsers implement standards inconsistently and make sure your site works reasonably well across all major browsers. At a minimum test against a recent Gecko engine (Firefox), a WebKit engine (Safari and some mobile browsers), Chrome, your supported IE browsers (take advantage of the Application Compatibility VPC Images), and Opera. Also consider how browsers render your site in different operating systems.

Staging: How to deploy updates without affecting your users. Have one or more test or staging environments available to implement changes to architecture, code or sweeping content and ensure that they can be deployed in a controlled way without breaking anything. Have an automated way of then deploying approved changes to the live site. This is most effectively implemented in conjunction with the use of a version control system (git, Subversion, etc.) and an automated build mechanism (Ant, NAnt, etc.).

Don't display unfriendly errors directly to the user.

Don't put users' email addresses in plain text as they will get spammed to death.

Add the attribute rel="nofollow" to user-generated links to avoid spam.

Build well-considered limits into your site - This also belongs under Security.

Learn how to do progressive enhancement.

Redirect after a POST if that POST was successful, to prevent a refresh from submitting again.

Read Don't Make Me Think.

Security Performance

Implement caching if necessary, understand and use HTTP caching properly as well as HTML5 Manifest.

Optimize images - don't use a 20 KB image for a repeating background.

Compress content for speed, see brotli, gzip/deflate (deflate is better).

Combine/concatenate multiple stylesheets or multiple script files to reduce the number of browser connections and improve gzip ability to compress duplications between files.

Take a look at the Yahoo Exceptional Performance site, lots of great guidelines, including improving front-end performance and their YSlow tool (requires Firefox, Safari, Chrome or Opera). Also, Google page speed (use with browser extension) is another tool for performance profiling, and it optimizes your images too.

Use CSS Image Sprites for small related images like toolbars (see the "minimize HTTP requests" point)

Use SVG image sprites for small related images like toolbars. SVG coloring is bit tricky. You can read about it here.

Busy web sites should consider splitting components across domains. Specifically...

Static content (i.e. images, CSS, JavaScript, and generally content that doesn't need access to cookies) should go in a separate domain that does not use cookies, because all cookies for a domain and its subdomains are sent with every request to the domain and its subdomains. One good option here is to use a Content Delivery Network (CDN), but consider the case where that CDN may fail by including alternative CDNs, or local copies that can be served instead.

Minimize the total number of HTTP requests required for a browser to render the page.

Choose a template engine and render/pre-compile it using task-runners like gulp or grunt

Make sure there's a favicon.ico file in the root of the site, i.e. /favicon.ico. Browsers will automatically request it, even if the icon isn't mentioned in the HTML at all. If you don't have a /favicon.ico, this will result in a lot of 404s, draining your server's bandwidth.

SEO (Search Engine Optimization)

Use "search engine friendly" URLs, i.e. use example.com/pages/45-article-title instead of example.com/index.php?page=45

When using # for dynamic content change the # to #! and then on the server $_REQUEST["_escaped_fragment_"] is what googlebot uses instead of #!. In other words, ./#!page=1 becomes ./?_escaped_fragments_=page=1. Also, for users that may be using FF.b4 or Chromium, history.pushState({"foo":"bar"}, "About", "./?page=1"); Is a great command. So even though the address bar has changed the page does not reload. This allows you to use ? instead of #! to keep dynamic content and also tell the server when you email the link that we are after this page, and the AJAX does not need to make another extra request.

Don't use links that say "click here". You're wasting an SEO opportunity and it makes things harder for people with screen readers.

Have an XML sitemap, preferably in the default location /sitemap.xml.

Use when you have multiple URLs that point to the same content, this issue can also be addressed from Google Webmaster Tools.

Use Google Webmaster Tools and Bing Webmaster Tools.

Install Google Analytics right at the start (or an open source analysis tool like Piwik).

Know how robots.txt and search engine spiders work.

Redirect requests (using 301 Moved Permanently) asking for www.example.com to example.com (or the other way round) to prevent splitting the google ranking between both sites.

Know that there can be badly-behaved spiders out there.

If you have non-text content look into Google's sitemap extensions for video etc. There is some good information about this in Tim Farley's answer.

Technology

Understand HTTP and things like GET, POST, sessions, cookies, and what it means to be "stateless".

Write your XHTML/HTML and CSS according to the W3C specifications and make sure they validate. The goal here is to avoid browser quirks modes and as a bonus make it much easier to work with non-traditional browsers like screen readers and mobile devices.

Understand how JavaScript is processed in the browser.

Understand how JavaScript, style sheets, and other resources used by your page are loaded and consider their impact on perceived performance. It is now widely regarded as appropriate to move scripts to the bottom of your pages with exceptions typically being things like analytics apps or HTML5 shims.

Understand how the JavaScript sandbox works, especially if you intend to use iframes.

Be aware that JavaScript can and will be disabled, and that AJAX is therefore an extension, not a baseline. Even if most normal users leave it on now, remember that NoScript is becoming more popular, mobile devices may not work as expected, and Google won't run most of your JavaScript when indexing the site.

Learn the difference between 301 and 302 redirects (this is also an SEO issue).

Learn as much as you possibly can about your deployment platform.

Consider using a Reset Style Sheet or normalize.css.

Consider JavaScript frameworks (such as jQuery, MooTools, Prototype, Dojo or YUI 3), which will hide a lot of the browser differences when using JavaScript for DOM manipulation.

Taking perceived performance and JS frameworks together, consider using a service such as the Google Libraries API to load frameworks so that a browser can use a copy of the framework it has already cached rather than downloading a duplicate copy from your site.

Don't reinvent the wheel. Before doing ANYTHING search for a component or example on how to do it. There is a 99% chance that someone has done it and released an OSS version of the code.

On the flipside of that, don't start with 20 libraries before you've even decided what your needs are. Particularly on the client-side web where it's almost always ultimately more important to keep things lightweight, fast, and flexible.

Bug fixing

Understand you'll spend 20% of your time coding and 80% of it maintaining, so code accordingly.

Set up a good error reporting solution.

Have a system for people to contact you with suggestions and criticisms.

Document how the application works for future support staff and people performing maintenance.

Make frequent backups! (And make sure those backups are functional) Have a restore strategy, not just a backup strategy.

Don't forget to do your Acceptance Testing. Frameworks like Selenium can help. Especially if you fully automate your testing, perhaps by using a Continuous Integration tool, such as Jenkins.

When logging make sure you capture both handled exceptions, and unhandled exceptions. Report/analyse the log output, as it'll show you where the key issues are in your site.

Other

Implement both server-side and client-side monitoring and analytics (one should be proactive rather than reactive).

Use services like UserVoice and Intercom (or any other similar tools) to constantly keep in touch with your users.

Follow Vincent Driessen's Git branching model

Lots of stuff omitted not necessarily because they're not useful answers, but because they're either too detailed, out of scope, or go a bit too far for someone looking to get an overview of the things they should know. Please feel free to edit this as well, I probably missed some stuff or made some mistakes.

# data[].attributes.attachment_link.title
What technical details should a programmer of a web application consider before making the site public?

# data[].attributes.attachment_link.description
What things should a programmer implementing the technical details of a web application consider before making the site public? If Jeff Atwood can forget about HttpOnly cookies, sitemaps, and cros...

# data[].attributes.attachment_link.readable_text
Table of contents

Introduction

There are a number of technologies we use everyday when we browse the web. There is the data itself (the web pages) obviously, the formatting of this data, the transport mechanism which allows us to retrieve this data, and there is the foundation, the root, the thing that makes the web a web: links from one page to the other. These links are URLs.

General URL syntax

Everyone by now has seen a URL at least once in his life. Take "http://www.google.com" for instance. This is a URL. A URL is a Uniform Resource Locator and is really a pointer to a web page (in most cases). URLs actually have a very well-defined structure since the first specification in 1994.

We can extract detailed information about the "http://www.google.com" URL:

If we look at a more complex URL such as "https://bob:bobby@www.lunatech.com:8080/file;p=1?q=2#third" we can extract the following information:

The Scheme (here http and https (secure HTTP)) define the structure of the rest of the URL. Most internet URL schemes have a common first part which indicates the user, password, host name and port, followed by a scheme-specific part. This common part deals with authentication and being able to know where to connect in order to request data.

HTTP URL syntax

For HTTP URLs (with the http or https schemes), the scheme-specific part of the URL defines the path to the data, followed by an optional query and fragment.

The path part consists in a hierarchical view similar to a file system hierarchy with folders and files. The path starts with a "/" character, then each folder is separated from one another by a "/" again until we reach the file. For example "/photos/egypt/cairo/first.jpg" has four path segments: "photos", "egypt", "cairo" and "first.jpg", which can be extrapolated as: the "first.jpg" file in the "cairo" folder, which is in the "egypt" folder located in the "photos" folder at the root of the web site.

Each path segment can have optional path parameters (also called matrix parameters) which are located at the end of the path segment after a ";", and separated by ";" characters. Each parameter name is separated from its value by the "=" character like this: "/file;p=1" which defines that the path segment "file" has a path parameter "p" with the value "1". These parameters are not often used - let's face it - but they exist nonetheless, and we've even found a very good justification for their use on a Yahoo RESTful API documentation:

Matrix parameters enable the application to retrieve part of a collection when calling an HTTP GET operation. See Paging a Collection for an example. Because matrix parameters can follow any collection path segment in a URI, they can be specified on an inner path segment.

After the path segments we can find the query which is separated from the path with a "?" character, and contains a list (separated by "&") of parameter names and values separated by "=". For example "/file?q=2" defines a query parameter "q" with the value "2". This is used a lot when submitting HTML forms, or when calling applications such as Google search.

Last in an HTTP URL is the fragment which is used to refer not to the whole HTML page but to a specific part within that file. When you click on a link and the browser automatically scrolls down to display a part which was not visible from the top of the page, you have clicked a URL with a fragment part.

URL grammar

The http URL scheme was first defined in RFC 1738 (actually even before in RFC 1630) and while the http URL scheme has not been redefined later the whole URL syntax has been generalised into Uniform Resource Identifiers (URIs) from a specification which has been extended a few times to accommodate for evolutions.

There is a grammar which defines how URLs are assembled, and how parts are separated. For instance, the "://" part separates the scheme from the host part. The host and path fragments parts are separated by "/", while the query part follows a "?". This means that certain characters are reserved for the syntax. Some are reserved for all URIs, while some are only reserved for specific schemes. All reserved characters that are used in a part where they are not allowed (for instance a path segment - a file name for example - which would contain a "?" character) must be URL-encoded.

URL-encoding is the transformation of a character ("?") into a harmless representation of this character which has no syntactic meaning in the URL. This is done by converting the character into a sequence of bytes in a specific character encoding, then writing these bytes in hexadecimal preceded by "%". A question mark in URL-encoding is therefore "%3F".

We can write a URL pointing to the "to_be_or_not_to_be?.jpg" image as such: "http://example.com/to_be_or_not_to_be%3F.jpg" which makes sure that nobody would think there might be a query part in there.

Most browsers nowadays display the URLs by decoding (converting percent-encoded bytes back to their original characters) them first, while keeping them encoded when fetching them for the network. This means users are almost never aware of such encoding.

Developers, or web page authors on the other hand have to be aware of it, because there are many pitfalls.

Common pitfalls of URLs

If you are working with URLs, it pays to know some of the most common traps you should avoid. Here we give a non-exhaustive list of some of those traps.

Which character encoding?

URL-encoding does not define any particular character encoding for percent-encoded bytes. Generally ASCII alphanumeric characters are allowed unescaped, but for reserved characters and those that do not exist in ASCII (the French "œ" from the word "nœud" - "knot" - for instance), we have to wonder which encoding to use when converting them to percent-encoded bytes.

Of course the world would be easier if there were just Unicode, because every character exists in this set, but this is a set - a list if you will - and not an encoding per se. Unicode can be encoded using several encodings such as UTF-8 or UTF-16 (there are several others), but then the problem is still there: which encoding should URLs (generally URIs) use?

The standards do not define any way by which a URI might specify the encoding it uses, so it has to be deduced from the surrounding information. For HTTP URLs it can be the HTML page encoding, or HTTP headers. This is often confusing and a source of many errors. In fact, the latest version of the URI standard defines that new URI schemes use UTF-8, and that host names (even on existing schemes) also use this encoding, which really rouses my suspicion: can the host name and the path parts really use different encodings?

The reserved characters are different for each part

Yes they are. Yes they are. Yes they are. (Yes they are).

For HTTP URLs, a space in a path fragment part has to be encoded to "%20" (not, absolutely not "+"), while the "+" character in the path fragment part can be left unencoded.

Now in the query part, spaces may be encoded to either "+" (for backwards compatibility: do not try to search for it in the URI standard) or "%20" while the "+" character (as a result of this ambiguity) has to be escaped to "%2B".

This means that the "blue+light blue" string has to be encoded differently in the path and query parts: "http://example.com/blue+light%20blue?blue%2Blight+blue". From there you can deduce that encoding a fully constructed URL is impossible without a syntactical awareness of the URL structure.

Suppose the following Java code to construct a URL:

Encoding the URL is not a simple iteration of characters in order to escape those that fall outside of the reserved set: we have to know which reserved set is active for each part we want to encode.

This means that most URL-rewriting filters would be wrong if they decide to take a URL substring from one part into another without proper encoding care. It is impossible to encode a URL without knowing about its specific parts.

The reserved characters are not what you think they are

Most people ignore that "+" is allowed in a path part and that it designated the plus character and not a space. There are other surprises:

"?" is allowed unescaped anywhere within a query part,

"/" is allowed unescaped anywhere within a query part,

"=" is allowed unescaped anywhere within a path parameter or query parameter value, and within a path segment,

":@-._~!$&'()*+,;=" are allowed unescaped anywhere within a path segment part,

"/?:@-._~!$&'()*+,;=" are allowed unescaped anywhere within a fragment part.

While this is slightly nuts and "http://example.com/:@-._~!$&'()*+,=;:@-._~!$&'()*+,=:@-._~!$&'()*+,==?/?:@-._~!$'()*+,;=/?:@-._~!$'()*+,;==#/?:@-._~!$&'()*+,;=" is a valid HTTP URL, this is the standard.

For the curious, the previous URL expands to:

Nuts.

A URL cannot be analysed after decoding

The syntax of the URL is only meaningful before it is URL-decoded: after URL-decoding, reserved characters may appear.

For example "http://example.com/blue%2Fred%3Fand+green" has the following parts before decoding:

Thus, we are looking for a file called "blue/red?and+green", not for the "red?and+green" file of the "blue" folder.

If we decode it to "http://example.com/blue/red?and+green" before analysis the parts would give:

This is clearly wrong: analysis of reserved characters and URL parts has to be done before URL-decoding. The implication is that URL-rewriting filters should never decode a URL before attempting to match it iff reserved characters are allowed to be URL-encoded (which may or may not be the case depending on you application).

Decoded URLs cannot be reencoded to the same form

If you decode "http://example.com/blue%2Fred%3Fand+green" to "http://example.com/blue/red?and+green" and proceed to encode it (even with an encoder which knows about each syntactical URL part) you will get "http://example.com/blue/red?and+green" because that is a valid URL. It just happens to be very different from the original URL we decoded.

Handling URLs correctly in Java

When you have mastered your black belt in URL-fu you will notice that there are still quite a few Java-specific pitfalls when it comes to URLs. The road to URL handling correctness is not for the faint of heart.

Do not use java.net.URLEncoder or java.net.URLDecoder for whole URLs

We are not kidding. These classes are not made to encode or decode URLs, as their API documentation clearly says:

Utility class for HTML form encoding. This class contains static methods for converting a String to the application/x-www-form-urlencoded MIME format. For more information about HTML form encoding, consult the HTML specification.

This is not about URLs. At best it resembles the query part encoding. It is wrong to use it to encode or decode entire URLs. You would think the standard JDK had a standard class to deal with URL encoding properly (part by part, that is) but either it is not there, or we have not found it, which lures a lot of people into using URLEncoder for the wrong purpose.

Do not construct URLs without encoding each part

As we have already stated: fully constructed URLs cannot be URL-encoded.

Take the following code for instance:

It is impossible to convert "http://example.com/a/b?c" back to what it should have been if "a/b?c" was meant to be a path segment, because it happens to be a valid URL. We have already explained this earlier.

Here is the proper code:

We are now using a utility class URLUtils which we had to make ourselves for lack of finding an exhaustive one available online fast enough. The previous code will give you the properly encoded URL "http://example.com/a%2Fb%3Fc".

Note that the same applies to the query string:

This will give you "http://example.com/?query=a&b==c" which is a valid URL, but not the "http://example.com/?query=a%26b==c" we wanted.

Do not expect URI.getPath() to give you structured data

Since once a URL has been decoded, syntactical information is lost, the following code is wrong:

It would first decode the path "a%2Fb%3Fc" into "a/b?c", then split it where it should not have been split into path segment parts.

The correct code of course uses the undecoded path:

Do note that path parameters will still be present: deal with them if required.

Do not expect Apache Commons HTTPClient's URI class to get this right

The Apache Commons HTTPClient 3's URI class uses Apache Commons Codec's URLCodec for URL-encoding, which is wrong as their API documentation mentions since it is just as wrong as using java.net.URLEncoder. Not only does it use the wrong encoder, but it also decodes each part as if they all had the same reserved set.

Fixing URL encoding at every level in a web application

We have had to fix quite a few URL-encoding issues in our application lately. From support in Java, down to the lower level of URL rewriting. We will here list a few of changes which were required.

Always encode URLs as you build them

In our HTML files, we replaced all occurrences of this:

with:

And similarly for query parameters.

Make sure your URL-rewrite filters deal with URLs correctly

Url Rewrite Filter is a URL rewriting filter we use in Seam to transform pretty URLs into application-dependent URLs.

For example, we use it to rewrite http://beta.visiblelogistics.com/view/resource/FOO/bar into http://beta.visiblelogistics.com/resources/details.seam?owner=FOO&name=bar. Obviously this involves taking some strings from one URL part to another, which means we have to decode from the path segment part and reencode as a query value part.

Our initial rule looked as follows:

It turns out that there are only two ways to deal with URL-decoding in Url Rewrite Filter: either every URL is decoded prior to doing the rule matching (the <to> patterns), or it is disabled and each rule has to deal with decoding. In our opinion the latter is the sanest option, especially if you move URL parts around, and/or want to match path segments which may contain URL-encoded path separators.

Within the replacement pattern (the <to> patterns) you can then deal with URL encoding/decoding using the inline functions escape(String) and unescape(String).

As of this writing, Url Rewrite Filter Beta 3.2 contains several bugs and limitations which blocked our progress towards URL-correctness:

URL decoding was done using java.net.URLDecoder (which is wrong),

the escape(String) and unescape(String) inline functions used java.net.URLDecoder and java.net.URLEncoder (which is not specific enough and will only work for entire query strings, beading in mind any "&" or "=" will not be encoded).

We therefore made a big patch fixing a few issues like URL decoding, and adding the inline functions escapePathSegment(String) and unescapePathSegment(String).

We can now write the almost correct:

It is only almost correct because our patch still lacks a few things:

the inline escaping/unescaping functions should be able to specify the encoding as either fixed (this is already done) or by determining it from the HTTP call (not supported yet),

the old escape(String) and unescape(String) inline functions were left intact and still call java.net.URLDecoder which is wrong as it will not escape "&" or "=",

we need to add more part-specific encoding/decoding functions,

we need to add a way to specify the decoding behaviour per-rule as opposed to globally in <urlrewrite>.

As soon as we get the time, we will send a second patch.

Using Apache mod-rewrite correctly

Apache mod-rewrite is an Apache web server module for URL-rewriting which we use to proxy all our http://beta.visiblelogistics.com/foo traffic to http://our-internal-server:8080/vl/foo for instance.

This is the last thing to fix, and just like Url Rewrite Filter, it defaults to decoding the URL for us, and reencoding the rewritten URL for us, which is wrong, as decoded URLs cannot be reencoded.

There is one way to get around this however, at least in our case since we are not switching one URL part for another, we do not need to decode a path part and reencode it into a query part for example: do not decode and do not reencode.

We accomplished it by using THE_REQUEST for URL-matching which is the full HTTP request (including the HTTP method and version) undecoded. We just take the URL part after the host, change the host and prepend the/vl prefix, and tada:

Conclusion

We hope to have clarified some URL myths and common mistakes. Short of clarifying them, making it obvious that it is not as simple as some people think will suffice. We have illustrated common mistakes in Java, and in the whole chain of a web application deployment. Now everyone is a URL expert and we hope never to see related bugs again. Please SUN, please: add standard support for URL encoding/decoding part by part.
Stéphane "FroMage" Épardaud is a senior software developer at Lunatech Research. He enjoys good arguments, especially if he will win them or/and learn something, so don't hesitate to comment by dropping him a mail.

# data[].attributes.attachment_link.title
What every web developer must know about URL encoding

# data[].attributes.attachment_link.description
There are a number of technologies we use everyday when we browse the web. There is the data itself (the web pages) obviously, the formatting of this data, the transport mechanism which allows us to retrieve this data, and there is the foundation, the root, the thing that makes the web a web: links from one page to the other.