Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Should HTML be encoded before being persisted?

Should HTML be encoded before being stored in say, a database? Or is it normal practice to encode on its way out to the browser?

Should all my text based field lengths be quadrupled in the database to allow for extra storage?

Looking for best practice rather than a solid yes or no :-)

like image 363
Razor Avatar asked Apr 11 '10 07:04

Razor


People also ask

Should I encode HTML?

You should always specify the encoding used for an HTML or XML page. If you don't, you risk that characters in your content are incorrectly interpreted. This is not just an issue of human readability, increasingly machines need to understand your data too.

When should data be encoded?

I always recommend to developers that they encode at the very last moment before they send the data to the external system.

How is HTML encoded?

HTML Encoding means to convert the document that contains special characters outside the range of normal seven-bit ASCII into a standard form. The type of encoding used is sent to the server in form of header information so that it can be easily and correctly parsed by the browsers.

How do I encode a HTML file?

Load the data to HTML–encode from a file, then press the 'Encode' button: Browse: Alternatively, type or paste in the text you want to HTML–encode, then press the 'Encode' button.


4 Answers

Is the data in your database really HTML or is it application data like a name or a comment that you just happen to know will end up as part of an HTML page?

If it's application data, I think its best to:

  • represent it in a form that native to the environment (e.g. unencoded in the database), and
  • make sure its properly translated as it crosses representational boundaries (encode when you generate the HTML page).

If you're a fan of MVC, this also helps separates the view/controller from the model (and from the persistent storage format).

Representation

For example, assume someone leaves the comment "I love M&Ms". Its probably easiest to represent it in the code as the plain-text String "I love M&Ms", not as the HTML-encoded String "I love M&Ms". Technically, the data as it exists in the code is not HTML yet and life is easiest if the data is represented as simply as accurately possible. This data may later be used in a different view, e.g. desktop app. This data may be stored in a database, a flat file, or in an XML file, perhaps later be shared with another program. Its simplest for the other program to assume the string is in "native" representation for the format: "I love M&Ms" in a database and flat file and "I love M&Ms" in the XML file. I would cringe to see the HTML-encoded value encoded in an XML file ("I love &Ms").

Translation

Later, when the data is about to cross a representation boundary (e.g. displayed in HTML, stored in a database, plain-text file, or XML file), then its important to make sure it is properly translated so it is represented accurately in a format native to that next environment. In short, when you go to display it on an HTML page, make sure its translated to properly-encoded HTML (manually or through a tool) so the value is accurately displayed on the page. When you go to store it in the database or use it in a query, use escaping and/or prepared statements and bound variable to ensure the same conceptual value is accurately represented to the database. When you go to store it in an XML file, you ensure its XML-encoded.

Failure to translate properly when crossing representation boundaries is the source of injection attacks such SQL-injection attacks. Be conscientious of that whenever you are working with multiple representations/languages (e.g. Java, SQL, HTML, Javascript, XML, etc).

--

On the other hand, if you are really trying to save HTML page fragments to the database, then I am unclear by what you mean by "encoded before being stored". If its is strictly valid HTML, all the necessary values should already be encoded (e.g. &, <, etc).

like image 177
Bert F Avatar answered Oct 26 '22 16:10

Bert F


The practice is to HTML encode before display.

If you are consistent about encoding before displaying, you have done a good bit of XSS prevention.

You should save the original form in your database. This preserved the original and you may want to do other processing on that and not on the encoded version.

like image 24
Oded Avatar answered Oct 26 '22 16:10

Oded


Database vendor specific escaping on the input, html escaping on the output.

like image 42
K. Norbert Avatar answered Oct 26 '22 16:10

K. Norbert


I disagree with everyone who thinks it should be decoded at display time, the chances of an attack occuring if its encoded before it reaches the database is only possible if a developer purposes decodes it before displaying it. However, if you decode it before presenting it there is always a chance that it could happen by some other newbie developer, like a new hire, or a bad implementation. If its sitting there unencoded its just waiting to pop out on the internet and spread like herpes. Losing the original data shouldnt be a concern. encode + decode should produce the same data every time. Just my two cents.

like image 31
user2175766 Avatar answered Oct 26 '22 15:10

user2175766