Real analytics from nginx logs, with no logging stack
Page views, top pages and per-site traffic, from the file already on your disk — no JavaScript, no third party, no Elasticsearch.
Every request your server handles is already written to a file. If all you need is page views, top pages and where traffic goes, you do not need a tracking script, a third-party account, or a log-shipping pipeline. You need to parse a file you already have.
First, log the one field that is missing
The default combined format is nearly enough, but it omits the thing that matters most on a server hosting more than one site: which host the request was for. Add it once, at the top of nginx.conf:
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" vhost=$host';
access_log /var/log/nginx/access.log main;Decide what counts as a view
This is where most home-made analytics goes wrong. A raw line count is not traffic. You want a definition and you want to apply it consistently:
- Not a bot. Match the user agent against a bot pattern, and treat an empty UA as a bot too.
- Not an asset. Exclude
.css,.js, images, fonts — a page with twelve images is one view, not thirteen. - Not an error. Only 2xx and 3xx. A 404 is not a page view.
- Not you. Drop your own IP and any private ranges.
The parse
re = r'^(\S+) \S+ \S+ \[([^\]]+)\] "(?:(\S+) )?([^"]*?)(?: HTTP/[\d.]+)?" '
r'(\d{3}) (\d+|-)(?: "[^"]*" "([^"]*)")?(?:.*\bvhost=(\S+))?'Group 4 is the path, 5 the status, 7 the user agent, 8 the vhost. Make the vhost group optional so the parser still works on a server that has not adopted the custom format — it degrades to "no per-site data" rather than to zero.
Cache the result, not the parse
Scanning a few hundred thousand lines takes a couple of seconds. That is fine for a cron job and far too slow for a page load. Write the aggregate to a JSON file with a timestamp and re-scan only when it goes stale:
cache = '/tmp/analytics-%d.json' % days
if fresh(cache, ttl=900):
return load(cache)
... scan ...
save(cache)Fifteen minutes of staleness is invisible to anyone reading a traffic report. Four seconds of page load is not.
What you get, and what you do not
You get page views, unique-ish visitors by IP, top pages, referrers, status-code breakdown, bandwidth, and per-site splits — all of it without a single byte of JavaScript on your pages and without sending anybody's browsing to a third party.
What you do not get: anything that requires client-side code. Time on page, scroll depth, viewport size, whether a visitor is the same person as yesterday. If you need those, you need a tracker. If you do not — and most people asking "how many people read that post" do not — the log is enough, and it is already there.
Alien XP
Runs the servers behind AlphaPanel and writes up what breaks.
