Why SQL Server seems to be so attached to transaction logs

Question

OK, here is goes. I’m not a database guru or admin. In fact, besides some occasional index / query tuning, I don’t poke around too often in databases. One of the things which often eludes me is the SQL Server transaction log. I know what it’s for, what it contains and how it works (at least conceptually), but I think I don’t get why SQL Server seems to be so attached to transaction logs.

Here is the first question. Correct me if I’m wrong, but it seems to me by default transaction logs will simply contains the entire history of all changes in the database. There are two indications that this is may be indeed the case. When I create a new database, the maximum size of its log is set to "unrestricted growth". The second reason is that I have often dealt with tiny databases with huge transaction logs which could not be shrunk no matter what I did. It seems to be so odd I cannot believe it’s true. Why would I ever want the entire history by default? All I care about is the latest version of the data in a consistent state. Well, I suspect that there may be valid reasons for it is some cases, but I would consider this as an extra option.

My second question is why is so complicated to get rid of a transition log? Is it just me, or is there really no direct way to do it? Just recently, I was trying to get rid of a 100MB+ log of a 5MB database, and the simplest way I found was to detach the database, delete the log and re-attach it again (and even that SQL Serve complained a bit). I tried the shrink command with all possible options I could find, but I was able to shrink in only to about 50%. The database was not in use (no active connections), and I honestly did not care about any past transitions at all. I noticed that there are possibly some other "ways" how to do it; some involving backups and restores.

I diligently tried to read the MSDN documentation and learn something more about transitions, but after about 15 minutes which felt like walking in mud in circles, I gave up. I know that to database admins and gurus my questions will sound silly. I appreciate any feedback.

Edit: After the first answers, I realized that I may not been clear enough. I’m aware how a transaction log works during transactions, and why it’s important, and that it can be used for backup purposes. I think I wanted to ask more from the developer point of view. Most of the time I deal with staging / test temporary databases which don’t need any backup, and which nobody is using except me, and I often find myself needing to transfer it and having a huge transitional log is an unnecessary inconvenience at that situation.

Answer 1

The database log is not some after the fact history record, like an IIS log. In databases with Write-Ahead Logging the log is the primary restore, redo and undo source, and for all purposes the authoritative source of data. Deleting or replacing the log is one of the worst decisions somebody administering a database can do. Your database may be corrupted at this moment.

The proper way to truncate the log is to do a full database backup, followed by a database log backup, followed by periodical log backups. This will free space in the log and allow for its reuse. It will not shrink the physical LDF file.

Another avenue is to change the recovery model to SIMPLE, which will allow the server to automatically recycle the log file as it sees fit. Changing the recovery model to SIMPLE has implications on the recoverability of the database as you won't be able to apply certain disaster recovery scenarios like point in time recovery nor will you be able to recover any data changes since last backup.

Understanding Logging and Recovery in SQL Server
Misconceptions around the log and log backups: how to convince yourself
Inside the Storage Engine: More on the circular nature of the log