HTML auswerten mit tidy

HTMLs lassen sich mit tidy und Standardwerkzeugen einfach bearbeiten.

Chef/Kunde/Kollegen: “Sag mir mal wie viele wir von X haben.” Diese Frage kommt in letzter Zeit, da wir in Mitten in der Analyse unserer Systeme sind. X kann dabei html-, pdf-, … -Dateien, 404er, ungültige Links, Nutzer, Zugriffe auf ….

Heute: Die Anzahl der aktiven Seiten eines Bereiches in unserem Intranet.

Aktive Seiten? Was ist aktiv? Unsere Plattform hat mehr als eine Quelle für Inhalte, HTML können an Navigationsknoten sein aber auch nicht. Sie können aber auch ohne Berechtigungen veröffentlicht sein, sind Sie dann aktiv? Der heutige Kompromis, da die Zahlen gestern benötigt waren, Auswertung der Sitemap für einen anonymen User.

Die Sitemap ist ein Einzler, aber enthält ungeordnete Listen mit den Navigationsknoten. Die Lösung soll in einem wohlgeordneten HTML die Navigationsknoten extrahieren, sodass pro Zeile genau einer ist, dann muss nur noch der relevante Teil ausgeschnitten und gezählt werden.

Das HTML bekommt man mit tidy wunderbar wieder in Form, die Navigationsknoten haben einen onClick Event und können dadurch erkannt werden, überflüssiges HTML wird entfernt, Bereich wird ausgeschnitten und die Zeilen gezählt.

tidy -iq -omit -utf8 -wrap 9999999 sitemap.html > sitemap.tidy
sed -n "/toggle/s/<strong.*\">//p" sitemap.tidy | cut -d "<" -f1 > sitemap.list
awk '/^Reisen/,/^Marktplatz/' sitemap.list
whitespace=$(($(head -1 sitemap.list | grep -oP '^\s+' | wc -c) - 1)) && gsed -i "s/^\s\{$whitespace\}//" sitemap.list
awk '/^Reisen/,/^Marktplatz/' sitemap.list | sed '$d'

Nach dem tidy sind die einzelnen Zeilen zwar sauber durch Leerzeichen eingerückt, aber die erste Zeile ist noch auch noch eingerückt. Diese werden in Variablen $whitespace gezählt und per sed entfernt.

Backup in the Sky with Mozy

Backup in the Sky with Mozy

the crash

I had some serious issues with my hard drive this year. One off my
backup drives was an external 3.5” Western Digital with 360 GB. Yes it
was, it died. It lived in a closet my to my desk for all it’s live,
but one day I got a larger one for replacement. So I cloned the drive
to the new one and relocated on top off my desk. My new MacBookPro was
happy to use it as a TimeMachine backup. Then when I cleaned up desk I
must have hit it (touched it slightly) and it fell over and died. The
drive made scratchy sounds. I think the read head got stuck between
the platters. Thanks to the new drive no data was lost. But this gave
me an unsafe feeling about my data.

no more external 3.5”

That disk could have been a bit more robust. Tipping over should not
be lethal, really. So I decided to never buy 3.5” external drives,
again. 2.5” drives are (hopefully) more designed to be carried around
and can survive minor hits.

more redundancy

A friend recently told me that he had problems with his external drive
where he save many of the kids pictures. It seemed to work
technically, he could also see it in his windows explorer, but there
was logical error, no access to the files. His wife brought the drive
to a local computer store, but they could not help. They do not have
any backups, since it all worked great so far. Loads of pictures of
their last years are gone. That’s the digital age. No more negatives
in some shoe box! For the next days my friends accident, my own some
month ago, repeatedly rumbled in my head. I could not get it out. To
free my head I decided to put one more layer in my backup strategy.

at home? remote? the cloud!

What do I want?

• I want to have another backup in case my home machine and my
  MacbookPro and my backup fails.
• I want to protect very important that just aren’t allowed to be lost.
• I want to be hassle free. Automatically!
• I want convenient restores.

Loosing all your hardware seems a bit artificial, but someone could
get into my apartment and take away everything. Or the house would
burn down, while all my stuff is there. One might think this is like
winning a lottery with a huge jackpot. But people win and people loose
data. To can not prepare to be a lottery winner, but you a avoid being
a data loser.

Backup need to happen without interaction. Backups that need to be
started manually do not work for me. There are many times where I will
defer them and not do them at all. When it comes to restoring apps
like TimeMachine show how simple it can be to get files back.
Versioning is also great, but not all files are versioned anyway. My
pictures and mp3s collections just grows, the individual files do not
change. Same with scanned documents.

What options do I have?

• I can buy another drive make backups to it and store it somewhere
  saver than my desk or places my kids could get to.
• I can buy that drive, make backups and bring them to a friend.
• I make some VPN backup solution to a friends computer / server /
  cloud.
• I can use a cloud service.

Since I want it to be without interaction, another drive doesn’t do
the job. Setting up an VPN backup could be a try, but I setting up
something myself, would leave me in a state where would always rethink
and improve the solution. That is not really what I want. It should
just happen. So it looks like I will try cloud services.

Carbonite vs. Mozy

To do some simple tests, I created an account for Carbonite and Mozy.

I installed Mozy first and gave it a try. It is free up to 2GB, no
time limitation that I am aware of. The UI is ok and browsing backups for restore shows me my mirrored
folder structure. Did some restore tries of sample files. I checked
out a couple of different versions, but they where all the same. Then
I realized that the time selector is not about versions of documents,
but points in time, like in time machine. That should be OK with
important non changing files.

Carbonite was second. The trial offer includes is a 14 day no limit
backup. Sounds good for a try, but since I am not a sniper when it
comes to buying something, 14 days are a bit short. The service comes
with a PreferencePane which is 32bit. This means that PreferencePane
always needs me to accept that it restarts. It do not like that. I do
not have it installed anymore, but the setup what to backup wasn’t as
transparent as mozy is. I did not really test the application /
service a lot. The 14 days were 14 days I had different stuff to do. I
fired Carbonite up a couple off times thought, but the PrefPane issue
got on my nerves.

Mozy

So the decision was easy. A spend money on a one year subscription for
unlimited storage with Mozy.

Let’s see how it performs and how I feel with it.

remove redundant entries in “open with” dialog

Until now I have not found out why Vmware Fusion creates multiple entries in my “open with” dialog (right-click in Finder // ctrl-click). But I found a solution that recreates the LaunchServices, which is the magic behind that right-click,

/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/Support/lsregister -kill  -r -domain local -domain system -domain user