What to do with the growing log files, how not to lose data and save disk space?

During the operation of Unix / Linux systems, administrators face one very widespread problem, but it is not as easy to solve as it seems at first glance.

In the course of work, system programs and services continuously generate a text report on their work. In each row of the report, the time mark of every event is specified with the accuracy of up to a second – it’s normal. In the event of system failures or incomprehensible behavior, the contents of these very reports are often the only clue left for masters.

Who and when logged in the system? Who logged in locally? Are there any failed remote login attempts, and from which IP address? Which commands were running on schedule? Have any processes failed?

In a properly configured system, all this data can be found in log files to understand what happened and when. It’s very convenient!

But you have to pay for everything. In this case, we have to sacrifice the disk subsystem and a minor (negligible on modern CPUs) increase in processing time.

In the meantime, the log files can grow so much that the server will simply be “paralyzed”. If you don’t take any preliminary measures, server shutdown won’t take long to happen. How to calculate the required disk space? You need to divide the free space on the disk by the growth rate of the log files.

Modern personal computers often feature at least 250 Gb of space and, usually, a small stream of records in the log files doesn’t affect disc capacity, so the problem of disc space almost never arises.

In the case with dedicated servers that use arrays of 4-6 SAS disks of 150 Gb and are hosting around a hundred sites with good traffic, the web server is likely to be the main source of logs. In addition, some web application developers may create logs for debugging purposes. Unfortunately, they often use web server subfolders instead of the standard/var/log/directory to have standard access to these files as well as to the entire site. I described one amusing example of solving the problem of one forgotten a log in article The History of a Forgotten Apache Log File.

In the case of virtual servers, the resources are mostly allocated for work, their amount is moderate, and depends on the tariff. With a cheap tariff, users don’t get much disk space (especially with an SSD). Consuming this volume within a week and having a blocked server is not a big deal with such a tariff.

Why wait for our server to stop? Let’s take measures, but first, learn how this problem was solved before us, and which of these solutions would be optimal in our cases.

Windows

In Windows, the logging system can be logically divided into the ones existing before Windows XP – a fast and understandable system, and the ones appearing with Windows Vista – use of XML in the log storage system.

If you were satisfied with the speed of viewing system events on Windows XP / 2003 on your computer 10 years ago, after the transition to Windows Vista and higher (7, 8, 8.1, 10), you should buy a new high-speed solid-state drive.

Otherwise, the whole power of convenience, expansion, detailing, and 75% XML redundancy will affect your computer and disk subsystem. Instead of getting a sneak peek into what exactly happened, we can see the following picture.

Imagine that Windows is slowly looking for the key to its cabinet in its pockets, opens its cabinet, slowly observes all his shelves, then slowly opens his folios, and slowly lays them on the table opening the page that interests you.

If you reach out to turn over the page, he will calmly pick up his volume and again, also slowly, will find the next page and put it on your table. Full of dignity, the system evokes respect – it works at its own pace, without interruptions.

You try not to disturb it.

But technical progress does not stand still, and the “old good man” who cannot support all the power of modern Microsoft ideas should not be overloaded with overwork. Modern equipment copes with this much better, and you don’t have to focus on such trifles as insufficient optimization and excessive (at first glance) resource intensity.

System journals are stored in files with evtx extension in the% SYSTEM_ROOT% \ System32 \ Winevt \ Logs \ directory.

The size of the system logs is regulated in the same directory where they are viewed – in the “Event Viewer” snap-in. Right-clicking on the journal line allows you to select the “Properties” menu item and the size policy.

As the rule, the system is set to maintain a fairly small file size. And if you have an actively used server, or a journal is growing faster than usual, in order not to lose records about recent events, you may want to increase the maximum size or even enable archiving.

Snap in the latest versions of windows presents a summary of the latest important events. It counts the number of errors, warnings, successful logins and authentication failures – as they say, “Pay, use and enjoy convenience.”

As for non-standard cases, one could recall % windir% \ WindowsUpdate.logbut it has already migrated to the system and became the part of history.

Unix / Linux

In the UNIX operating system, the log file storage system has historically come up with the following solutions:

  1. All log files are located in the /var/log directory or in its subfolders with the name of a service or program, for example, samba, Apache, MySQL, etc.
  2. Log files are simply text files readable by any program for viewing text.
  3. Each line begins with the time stamp of the event with the date and year indicated, and accuracy of a second or more.
  4. Each log file contains the name (or IP address) of the node from which the logs are received.
  5. If in the event log file more than 1 service/program is followed by a timestamp followed by the program name.
  6. Syslog system service was created and standardized, allowing program developers to not bother themselves with some minor logging issues affecting security as well and administrators to configure the system log collection parameters.
  7. If the device has very little or no storage at all, the standard syslog service provides for the collection of data via the network or the transfer of messages to a remote device over the network.
  8. Administrators can customize the detailing of logging programs from complete absence to a complete torrent of debugging messages for every minor failure of the program or service they’re concerned with.
  9. The system service logroate was created with the purpose of creating an archive of log files that renumbers the queue and deletes old logs.
  10. During formatting, the system allocates additional space for the processes of the superuser so that he could somehow connect and fix the problem of a full disk.

The items listed above are generally used primarily in official and trustworthy software products. As for bare-bones, amateur programs, they can write logs, but no one guarantees those will be stored in the / var /log/directory. There are some new trends, as well: in the systemd, point 2 is not executed, which means the logs of this program can only be read using the special software supplied with systemd. This is one offf many reasons why systemd is always criticized by many old-school admins. Let’s get through the other points of our list.

Regarding point 3: at the moment, there is no single standard for timestamp format. The main recommendation is to use one server in all logs, if possible, the same format. If possible, use the same time format and the same time zone on all servers.

6 – Syslog – There is an Internet standard described in RFC 5424. For open source operating systems, this service is implemented by more than one development team in different projects with different target systems. Some put a focus on the simplicity and speed of implementation and minimization of resources consumed. In others, the emphasis is placed on the flexibility of settings, extensibility of the functionality of plug-ins, connections to other systems for collecting operational data, and the built-in language for sorting events by files. Depending on the case, choose what is right for you.

7 – Network logging, for the UDP protocol, the protocol is described in RFC 5426, the default UDP port 514 is defined. For TCP, there is a description in RFC 6587 with the HISTORIC status. In RFC 5424, there is a description of the delivery of messages over the network using TPC, TLS, HTTPS b. In addition, a special protocol RELP is used. If you have a network that unites several (and maybe more) Unix servers, it is logical to select one common centralized logging server.

8 – Setting of logging detailing depends heavily on the specific software. The better a software product is developed, the more likely it is to feature appropriate settings for detailing and sending logs to files or the syslog system service. Some programs allow you to reconfigure these options “on the fly”, others will require a mandatory reboot, you need to take this into account and carefully read the documentation.

9. – Log rotation and logrotate program. The log rotation means division of the log into several files structures in ascending order from more recent to those that are older. Optionally, you can compress files in rotation to save space. In the configuration file, you can define:

  • Rotation conditions – file size exceeded or time elapsed.
  • Number of files in the archive.
  • How to rename files – you can add the date to the name.
  • Whether to use the compression algorithm for the archive.
  • Whether it is necessary to create an empty file in replacements processed and with which access attributes and who should be the owner so that the service can continue to write logs.
  • Additional programs or scripts that run for various program events. Before the operation – prerotate, after the operation – postrotate, before deletion – preremove.

Normally, logrotate is already installed in your distribution. Its configuration file or the catalog with configuration contains settings supplied with each package of such programs as syslog (rsyslog or syslog-ng), exim, dovecot, postfix, apache, nginx.

The distribution support team writes a standard configuration file which may need to be corrected if you have redefined the default settings for the corresponding program. Most modern programs that write logs directly to files, including the well-known Apache web server, need to be “warned” that the file into which they are writing the log has already been deleted or renamed and possibly compressed.

If you do not do this or do it incorrectly, the result will surprise you! The program will continue writing to the file, but it will not be in the file table anymore.

At the same time, the file system will honestly display the decreasing free space using the df utility or in Linux in pseudo-files in the proc and sys folders.

Meanwhile, summing the file sizes with ANY Most modern programs that write logs directly to files, including the well-known Apache web server, need to be “warned” that the file into which they are writing the log has already been deleted or renamed and possibly compressed.

The most annoying thing about it is that usually, services running on the server on that moment stop and display errors about insufficient disk space. Only reloading of the corresponding process can help. If you don’t know exactly which process causes the problem, you need to restart the entire server.

Many daemons of services are written in such a way that they process the HUP or USR signal just to rotate the logs. They close all open files, re-read configurations but do not break user network sessions: it’s done to justify the UNIX image as a reliable OS that does not require years of reboot.

Other programs may be easier to implement, they just need to be restarted. User scripts and Web applications that work when each page is generated will most likely open a file with logs each time and will not require any modifications.