LogReport Foundation; Annual Report 2001

tools for computer/network log file analysis

1. Introduction

This annual report consists of the following sections:

1. Introduction
2. Activities in 2001
3. Official Information
4. Financial Statements

1.1 The goal of LogReport

LogReport aims to transform the data in log files of network/computer systems into useful information. Log files are usually treated like the waste of Information Technology activity. Files are sitting somewhere in a dark corner of a computer system and get only touched occasionally in case of reactive problem solving. This is unfortunate because log files contain the traces of computer activity, and by systematic analysis of this activity one can learn a lot of the behavior of the systems and the users. Stichting LogReport Foundation develops and maintains Open Source reporting software and documentation.

1.2 LogReport basic idea

Log files are distilled into an intermediate format (also known as DLF = Distilled Log Format) from which the reports can be generated. The intermediate step introduces a log file dependent and a report dependent part. The log file dependent part translates to the DLF format and the report dependent part composes the report from the DLF files. The intermediate format can be used as the store of a data warehouse which allows you to generate many reports from many log files from many sources.

2. Activities in 2001

2.1 Introduction

We report on the activities of the LogReport team during 2001.

The main task of this team is maintenance and development of Lire, LogReport's tool for performing an integrated analysis of all ones Internet and Intranet Services. Lire does this by automatically generating useful reports from raw logfiles from various services. Next to this work, the team has been doing promotional activities like writing articles, giving presentations and maintaining the website. Finally, the team has been administering the LogReport server.

LogReport's server, called hibou.logreport.org, is hosted at VIA Networks in Eindhoven, The Netherlands. The machine hosts the LogReport website http://logreport.org/, handles LogReport email, and hosts the 4 public mailinglists:
<commit@logreport.org>, <announcement@logreport.org>, <questions@logreport.org>, and <development@logreport.org>. Furthermore, it hosts a CVS repository for version control of non-public documents and hibou's configuration data. The Lire code development is done using CVS on the SourceForge Open Source Development servers, http://sourceforge.net/projects/logreport/.

Various developers' meetings took place: On March 24th, there was a meeting at Jan Stap's place in Eindhoven, The Netherlands, attended by Joost, Jan and Egon. On May 8th, Joost, Egon and Wytze met at Joost's place in Eindhoven. On July 4th, Joost, Francis, Christoph and Egon met at the Libre Software Meeting in Bordeaux, France. All other developers' communication was done using email and IRC.

A more in-depth overview, specifically on Lire development, can be found in the variousJournal and Roadmapdocuments, which were presented at the board meetings in June, September and November 2001. A detailed list of the improvements on Lire can be found in the Lire NEWS file. Some highlights can be found on the LogReport history page on http://logreport.org/about/history.php too.

2.2 People

The year was started with Joost van Baal working for LogReport, 3 days a week. Furthermore, Anton Holleman, Peter Huisken, Gerrit Nijhuis and Jan Stap did work for the LogReport project on a volunteer basis. During the entire year, Teus Hagen, Wytze van der Raay and Jakob Schripsema formed the LogReport Foundation's board. Since only Joost could spent a large amount of time on LogReport work and Lire development, more contributors were welcome. In January, we started advertising vacancies. We've been talking to quite a few candidates.

In February 2001, Egon Willighagen from Nijmegen, The Netherlands joined the LogReport team. Especially his background in XML development made him suitable for Lire development.

On June 15, Francis J. Lacoste from Montréal, Canada and Christoph Lameter from Newark, California, USA, joined the LogReport team. Francis has a thorough background as a Perl hacker, Christoph knows how to set up a company; his TelemetryBox project, a Linux based distribution designed especially for network diagnostic purposes, could create some synergy with Lire.

On June 25, LogReport said goodbye to Anton, Peter en Gerrit as LogReport volunteers. Their fulltime jobs did not leave them sufficient time for LogReport work. Anton continues to be involved as external advisor to the board.

On August 23, Plamen Bozukov from Varna, Bulgaria, joined the LogReport team. He worked on, among other things, a Lire CGI interface.

On September 1, Christoph left LogReport: he started a fulltime job working on his TelemetryBox project.

On October 1, 2001 Joshua Koening from New York, USA and Wessel Dankers, from Bunnik, The Netherlands, joined the LogReport team. Wessel has been doing, among other things, PHP development. He is involved in several Linux development projects. Josh has a background as an internet and new media consultant. They both started their LogReport involvement by working on the LogReport website.

At that time, the LogReport team was up to its tasks. Hiring stopped.

Since November 1, Egon is no longer hired as an employee, but remains active as a volunteer. He decided to spent more time on his job as a bio-informatics researcher at Nijmegen University. He continues to do work for LogReport on an ad-hoc basis.

2.3 Lire Software Releases

In the year 2001, the following Lire versions have been released:

release date	version	filename(s)
January 16	20010116	lr-20010116.tar.gz
February 19	20010219	lr-20010219.tar.gz
March 18	Lire 20010318	lire-20010318.tar.gz
April 7	Lire 20010407	lire-20010407.tar.gz
April 18	Lire 20010418	(pointrelease for Debian package)
		lire-20010418.tar.gz,
		lire_20010418-1_all.deb
May 9	Lire 20010509	lire-20010509.tar.gz,
		lire_20010509-1_all.deb
June 26	Lire 20010626	lire-20010626.tar.gz
June 29	Lire 20010629
	and 20010629.1	(pointrelease)
		lire-20010629.1.tar.gz,
		lire_20010629.1-1_all.deb
September 3	Lire 20010903	lire-20010903.tar.gz
October 17	Lire 20011017	lire-full-20011017.tar.gz,
		lire-20011017.tar.gz
December 5	Lire 20011205	lire-full-20011205.tar.gz,
		lire-20011205.tar.gz
December 5	Lire 20011205.1	(pointrelease)
		lire-full-20011205.1.tar.gz,
		lire-20011205.1.tar.gz,
		lire_20011205.1-1_all.deb,
		lire-20011205-2.noarch.rpm,
		lire-html-20011205-2.noarch.rpm,
		lire-pdf-20011205-2.noarch.rpm

Only since March 18 the LogReport software is called Lire. Pointreleases differ only in minor aspects from previous releases, these were made mainly to facilitate packaging. Note that since April 19 we ship Lire as a Debian package too, to facilitate installation on Debian GNU/Linux, and since December 16, we also ship Lire as a RPM, to facilitate installation on RPM-based system, like Red Hat Linux.

A Debian package based on Lire 20010626 was shipped with Christoph Lameters' TelemetryBox 1.2.4 in June 2001.

We list the main improvements in the various releases, as posted on the LogReport Announcement list. A more detailed overview is in the NEWS file, as distributed with Lire.

20010116	The package no longer depends on GNU make. Lire is fully "autoconfiscated", which makes installation more flexible, and makes it easier to package Lire for various Unix distributions. Lire is ported to SUN Solaris. A "client" package is added, to be able to use Lire without installing the complete Lire suite.
20010219	Anlr_runscript is added. This gives more userfriendly errormessage handling, and an easier interface for debugging. A local site config filedefaults.localis added: improved support for local configuration. Some more www reports are added.
20010318	Lire now generates XML output. The userfriendly configuration scriptlr_configis added.
20010407	Support for exim MTA log files is added. The internal format to represent reports is moved to pure XML, enabling more flexible output layer handling.
20010509	Reports can now get typeset in Adobe PDF format. Lire's documentation is converted to DocBook XML.
20010629.1	Reports can be typeset in HTML. PDF reports, as well as HTML reports, now can include images and graphs.
20010903	Various improvements and enhancements in the www reporting are made. Lire supports new output formats: LogML (an experimental XML application) and DocBook XML.
20011017	Various improvements and enhancements in the www and email reporting are made. Cisco ACL log files are now supported, as well as xferlog FTP log files. New supported output formats are XHTML and RTF.
20011205	CUPS (page_log) and LPRng (lp-acct) print log files are supported. W3C Extended Log Format webserver logs, as used by Microsoft's Internet Information Server is supported. Linux 2.2 ipchains firewall log, as well as Darren Reed's IP Filter log as shipped with *BSD's, are supported. New www, ftp and email reports are added.

2.4 Statistics

We show some statistics, giving some indications on the work done by the LogReport team, as well as on the effectiveness of this work.

2.4.1 Downloads and installs

We do have figures about the number of downloads from our webserver. They're depicted in the next graph.

Number of Lire tarball downloads from
http://logreport.org/
in 4-week periods

Based on the Debian Popularity Contest, we estimate the number of people who've installed the Lire Debian package to about 200 at the end of 2001. (This was 0 until July: at that time there was no Debian Lire package yet.)

Please note that Lire is also available for download directly from Sourceforge. The Sourceforge-supplied statistics show that the number of downloads grew from 0 during January 2001 to 39 during April 2001 to 581 during October 2001.

2.4.2 Submits to the LogReport Online Responder

During the entire year, we've been running a so called Online Responder on <log@<service>.logreport.org>. People can submit their (anonymized and/or compressed) log files via email, and get a report back.

In July, we had about 4 different submitters, in August we had about 10 different submitters, in September about 40, in October about 60, in December about 50. At the end of the year, we had about 6 regular submitters, who submit their logs and keep coming back for a long time.

2.4.3 CVS Commits to the Lire code

There were between 100 and 700 commits on the Concurrent Version System holding the Lire code done monthly during 2001. The total number of commits was 5496. This number represents the number of changes to the Lire code.

2.4.4 Traffic on the LogReport mailing lists

During the year, the number of posts on our mailinglists has been increasing. However, we didn't succeed yet in building an impressively actively contributing user community. We did succeed in laying the seeds, though: some small contributions were made.

At the end of the year, about once a week, an external interested person posts to any of LogReport's mailing lists.

2.5 Cooperation with developers

2.5.1 External developers

A few people suggested improvements, using our mailinglists.

In July 2001, the LogReport team was contacted by John Punin, about the possibility of Lire interfacing with LOGML, an XML 1.0 application designed to describe log reports of web servers. Egon coded Lire support for this. Jesus M. Salvo Jr., the OpenJGraph maintainer, was contacted about tools to make the generated data easily accessible.

In August 2001, Egon contacted AWStats maintainer Laurent Destailleur about including his code in Lire. There were no real obstacles, since AWStats is free software. Synergy was achieved.

Joost Bekkers contributed valuable code to Lire, including convertors for proxy and cisco log files.

Since December 2001, Cédric Gross has been active on our mailinglists.

2.5.2 Licensing debate

The LogReport team has worked on improving external cooperation by clarifying the copyright ownership issue. A discussion took place among <people@logreport.org> (Joost van Baal, Plamen Bozukov, Francis Lacoste, Christoph Lameter, Jan Stap, and Egon Willighagen and <bod@logreport.org> (Teus Hagen, Jakob Schripsema and Wytze van der Raay) from mid July until the end of the year.

The outcome of this discussion has been layed down in the document titledCopyright and License of code, distributed by LogReport, as well as in the Licensing section of theFrequently Asked Question About Liredocument, as distributed with Lire. So, we're now able to respond in a professional way when someone wants to contribute code.

2.6 Website

On December 8, a totally revised LogReport website went life. Webmasters Joshua Koenig and Wessel Dankers did an excellent job here: the look and feel of the website is much more professional, while the backend is restructured into more maintainable code.

2.7 Promotion

On April 26, Joost did an improvised LogReport session at Moscow State University, which was attended by some Russian Computer Science students.

On November 30, Egon gave a presentation at Linux Kongress in Enschede, The Netherlands. An article on Lire is featured in the conference proceedings.

The September and November issues of LinuxFocus feature an article in two parts about Lire by Egon.

2.8 LogReport Future

On November 15 2001 Joost reported to the developers about the November Board Meeting. At that meeting, it was made clear once again, that LogReport funding is uncertain after the end of 2002. A long discussion has started since on how LogReport should proceed after 2002, and this discussion is continuing to date.

2.9 Contact information

web	http://www.logreport.org/
email	<logreport@logreport.org>
irc	#logreporton the OpenProjects.net IRC network
sourceforge	http://sourceforge.net/projects/logreport/
snailmail	Stichting LogReport Foundation
	p/a J.E. van Baal
	Strijpsestraat 48-A
	5616 GR Eindhoven
	The Netherlands

3. Official Information

Stichting LogReport Foundation has been established on August 21, 2000 in Eindhoven, The Netherlands. The goal of the foundation is:

1. to develop, maintain and distribute tools and knowledge for processing log files of network/computer system applications and for generating reports based on such log files;
2. to stimulate the use of the tools and knowledge mentione above for the management of information systems;
3. to stimulate authors of network/computer system applications to incorporate provisions in these applications for generating useful standardised and automatically processable information in log files;
4. to contribute to the development and implementation of product-independent log file formats (standards);
5. to create a forum for system administrators and software developers in the area of the application and analysis of log file information; and
6. anything which is directly or partly related to the above, or can be beneficial to the above, in the widest sense.

3.1 Board

The board of Stichting LogReport Foundation consists of three members:

Teus Hagen	chairman
Jakob Schripsema	secretary
Wytze van der Raay	treasurer

The composition of the board has not been changed since its start on August 21, 2000.

Four board meetings were held in 2001:

	date	place
	April 12, 2001	Oss
	June 13, 2001	Oss
	September 12, 2001	Oss
	November 14, 2001	Oss

Generally, board meetings have also been attended by Joost van Baal (liaison with LogReport operations) and Anton Holleman (advisor to the board).

3.2 Employees

The foundation has employed on average two part-time staff members during the year 2001:

Joost van Baal	November 2000 - now	software developer
Egon Willighagen	February 2001 - October 2001	software developer
Wessel Dankers	October 2001 - now	webmaster

In addition, a number of freelancers abroad have also performed paid remote development work for the foundation:

Christoph Lameter	USA	June 2001 - August 2001
Francis Lacoste	Canada	June 2001 - now
Plamen Bozukov	Bulgaria	September 2001 - December 2001
Joshua Koenig	USA	October 2001 - now

Finally, a number of volunteers have performed or are performing various tasks for the foundation.

3.3 Administration

The responsibility for day-to-day administration, handling of payments and other similar activities which was initially assigned to Anton Holleman, has been transferred in the first half of 2001 to Joost van Baal. The bookkeeping function has been taken care of by Wytze van der Raay, treasurer of the foundation. The salary administration and related work (interfacing to GAK, ARBO service etc) has been contracted out to De Wert Accountants in Eindhoven.

3.4 Fiscal year

The fiscal year of Stichting LogReport Foundation coincides with the calender year. Thus the second fiscal year of the foundation ran from January 1, 2001 until December 31, 2001. Therefore comparison with numbers from 2000, the first, incomplete, book year of the foundation (August 21, 2000 - December 31, 2000), does not always yield meaningful results, unfortunately.

3.5 Fiscal position

Based on its current activities, the foundation is not taxable for Dutch corporate tax ("vennootschapsbelasting") or value-added tax ("BTW").
On November 28, 2000, Stichting LogReport Foundation has been recognised by the Dutch fiscal authories as an organisation working for the general benefit ("algemeen nut beogende instelling") as meant in article 24, paragraph 4 of the Dutch Inheritance Act 1956.

4. Financial Statements

4.1 Balance Sheet per December, 31 2001 (after result allocation)

	2001			2000
	€	€		€	€
Fixed assets
Material fixed assets
Computer equipment		1,853.63			2,717.55
Current assets
Accounts receivable
Interest to be received		728.00			0.00
Costs paid in advance		590.72			458.55
Cash		54,080.27			2,261.89
		57,252.62			5,437.99
Own capital		47,194.64			2,446.25
Short-term liabilities
Accounts payable	4,574.85			303.32
Taxes and social charges	3,600.18			2,094.75
Other liabilities	1,882.95			593.67
		10,057.98			2,991.74
		57,252.62			5.437.99

4.2 Profit and Loss Account 2001

	2001			2000
	€	€		€	€
Other income
Donations received		140,000.00			16,500.00
Other expense
Payroll expenses	44,595.63			5,425.05
Depreciation of material
fixed assets	1,402.92			220.34
Other operational expenses	50,048.83			8,413.81
		96,047.38			14,059.20
		43,952.62			2,440.80
Interest earned	795.77			5.45
Interest paid	0.00			0.00
		795.77			5.45
Gross result from regular
operations before tax		44,748.39			2,446.25
Tax		0.00			0.00
Net result		44,748.39			2,446.25

4.3 General explanations

4.3.1 Basis for valuation and result determination

Assets and liabilities
Unless stated otherwise, assets and liabilities have been stated at their nominal values.

Material fixed assets
Material fixed assets have been stated at historical cost price, reduced with depreciation calculated linearly based on the estimated total useful life of the corresponding fixed asset.

4.3.2 Result determination

General
The following holds with respect to items included in the operational result: profits are only included if and for the part they have been realized in the reporting period, and losses and risks have been taken into account inasmuch they originate before the end of the reporting period.

4.4 Amplification of the Balance Sheet 2001

4.4.1 Material fixed assets

The material fixed assets can be specified as follows:

	Inventory	Computer-	Total
		equipment
	€	€	€
January 1, 2001
Procurement costs	0.00	2,937.89	2,937.89
Cumulative depreciation	0.00	(220.34)	(220.34)
Book value	0.00	2,717.55	2,717.55
Changes
Desinvestments	0.00	0.00	0.00
Investments	0.00	539.00	539.00
Depreciation	0.00	(1,402.92)	(1,402.92)
Depreciation desinvestments	0.00	0.00	0.00
		(863.92)	(863.92)
December 31, 2001
Procurement costs	0.00	3,476.89	3,476.89
Cumulative depreciation	0.00	(1,623.26)	(1,623.26)
Book value	0.00	1,853.63	1,853.63
Depreciation percentage	20%	50%

4.4.2 Accounts receivable

Accounts receivable consist of an estimate of the interest to be received on the savings account (due per May 1, 2002), an amount of expenses paid in advance for the months January and February 2002 with respect to a hosting contract with VIA NET.WORKS, an amount of expenses paid in advance for the period up to November 2002 with respect to ARBO service, and an amount of expenses paid to Gandi, Paris for domain name owner changes (to be effectuated in 2002).

4.4.3 Cash

Cash is kept on a business giro account and an associated savings account ("Kapitaalrekening") at Postbank N.V.

	2001		2000
	€		€
Postbank giro account	7,777.85		2,261.89
Postbank Kapitaalrekening	46,302.42		0.00
	54,080.27		2,261.89

4.4.4 Own capital

Reserve fund
The course is as follows:

	2001		2000
	€		€
Value per January 1	2,446.25		0.00
Plus: net result	44,748.39		2,446.25
Value per December 31	47,194.64		2,446.25

4.4.5 Short-term liabilities

Taxes and social charges

	2001		2000
	€		€
Salary tax	3,046.08		1,233.56
Social charges bedrijfsvereniging	554.10		861.19
	3,600.18		2,094.75

Other liabilities

	2001		2000
	€		€
Salary administration expenses	128.52		181.29
Holiday allowances	1,754.50		412.36
Savings salary	0.01		0.02
Net salary	(0.08)		0.00
	1,882.95		593.67

4.4.6 Liabilities not shown in the balance sheet

The foundation has engaged in a year contract for internet hosting services with VIA NET.WORKS; the costs of this contract are currently € 2,851.20 per year.

4.5 Amplification of the Profit and Loss Account 2001

4.5.1 Income

Income over 2001 consists solely of four quarterly donations from Stichting NLnet with a total value of € 140,000.00.

4.5.2 Payroll expenses

	2001		2000
	€		€
Staff salaries	38,961.15		4,698.46
Social charges	5,634.48		726.59
	44,595.63		5,425.05
Staff salaries
Salaries	36,036.09		4,260.00
Salary tax on savings salary	118.18		13.12
Holiday allowances	2,806.88		340.80
Overhevelingstoeslag	0.00		84.54
	38,961.15		4,698.46
Social charges
Social charges	5,550.90		726.59
ARBO service	83.58		0.00
	5,634.48		726.59

4.5.3 Other operational expenses

	2001		2000
	€		€
Other staff expenses	42,331.95		376.14
Office expenses	5,683.85		1,175.50
General expenses	2,033.03		6,862.17
	50,048.83		8,413.81
Other staff expenses
Travel expenses staff	1,740.73		74.86
Travel expenses volunteers	225.22		301.28
Compensation volunteers	76.00		0.00
Cost of freelancers	40,290.00		0.00
	42,331.95		376.14

	2001		2000
	€		€
Office expenses
Housing expenses	413.77		0.00
Office materials	38.62		44.92
Forwarding charges	16.51		123.88
Telephone costs	1,038.22		0.00
Internet expenses	3,716.43		995.38
Computer expenses	460.30		11.32
	5,683.85		1,175.50
General expenses
Professional books	321.63		0.00
Subscriptions & contributions	258.56		35.62
Conferences/courses	90.76		192.86
Publicity	22.94		0.00
Accountancy fees	506.52		181.29
Consulting fees	0.00		6,273.02
Travel expenses board	345.51		172.58
Other board expenses	279.52		0.00
Representation costs	22.23		0.00
Bank expenses	185.50		6.80
Calculation differences	(0.02)		0.00
Euro-calculation differences	(0.12)		0.00
	2,033.03		6,862.17

4.5.4 Interest earned

	2001		2000
	€		€
Credit interest Postbank giro account	67.77		5.45
Credit interest Postbank Kapitaalrekening	728.00		0.00
	795.77		5.45

4.6 Allocation of net result 2001

In anticipation of a board decision to be taken, the net result over 2001 ad € 44,748.39 has been added to the reserve fund.