It's a good idea to check what manually installed packages you have on your system every now and then (vs packages that were automatically pulled in as dependencies).

Every extra bit of software installed is a potential additional attack vector for nefarious minds, and more log or alert messages that you'll receive when your system tells you about them being updated for you. (Spam sucks, but more importantly it de-sensitizies you to things you should be paying attention to!)

For the sensible ones using apt:

comm -23 \
  <(apt-mark showmanual | sort -u) \
  <(gzip -dc /var/log/installer/initial-status.gz | sed -n 's/^Package: //p' | sort -u)

...or those weirdos using aptitude:

comm -23 \
  <(aptitude search '~i !~M' -F '%p' | sed "s/ *$//" | sort -u) \
  <(gzip -dc /var/log/installer/initial-status.gz | sed -n 's/^Package: //p' | sort -u)

You may find that you don't remember why you installed some libraries. It's not uncommon for people to needlessly manually install libraries that would have automatically been installed as dependencies.

You can check to see if any given package is depended on (is a dependency of) any other installed packages using the following commands:

Using apt:

apt-cache --installed rdepends libhtml-template-perl
libhtml-template-perl
Reverse Depends:
  mysql-server-5.5
  mysql-server-5.5
  libparse-debianchangelog-perl

Using aptitude:

aptitude why libhtml-template-perl
i   mysql-server-5.5 Recommends libhtml-template-perl

Once you find out you needlessly manually installed a library, then you can remark those packages explicly as "auto" instead of "manual" installs:

Using apt:

apt-mark auto libhtml-template-perl

Using aptitude:

aptitude markauto libhtml-template-perl