50-100 websites<\/strong> of diverse subject matters, it was discovered that an administrator forgot to enable the log rotation of a single website.<\/p>\n\n\n\nForgot to do it a couple or so years ago. Time passed by, the server worked, as usually, there was plenty of space on modern disks, and the log grew (like a turnip from the children’s fairy tale). When the log caught my eye, I didn’t even want to believe my own eyes.<\/p>\n\n\n\n
The file was huge; it occupied more than 135 GB<\/strong>. 135 gigabytes !<\/strong> of the browsing history. At the same time, less than 40 GB of free space remained at the section. By normal practice, a solution to such an omission is trivial and simple \u2013 to delete the oversized file, to send a signal to apache to re-open the files, which will cheerfully continue to write reports about its work to a newly created file; of course, you shouldn’t forget to enable the rotation setting for this file.<\/p>\n\n\n\nBut sometimes, website owners and web developers badly need the information from these very logs. And keeping the Murphy’s law and my previous experience of deleting supposedly useless and suddenly extremely important files in mind, I took on the task of dividing and compressing the file as if the rotation had been enabled, for example, on the monthly basis. A couple of years of logs would look like 24 + N files of compressed archives and 1 file of acceptable size.<\/p>\n\n\n\n
First of all, I looked through the standard Linux utilities aimed at something like this; chances are that I’m not the first one to face a similar task and there is a solution \u2013 all I had to do is to find it.<\/p>\n\n\n\n
Let’s start with the simplest commands and then move on to the “heavy staff”:<\/strong><\/p>\n\n\n\n- tail<\/strong><\/li><\/ul>\n\n\n\n
– tail -n 1500 BIG_BIG_LOGFILE.log<\/span> would show the last 1500 lines.<\/p>\n\n\n\n-n 1500 BIG_BIG_LOGFILE.log | gzip -c > log_tail1500.log.gz<\/span> will create a compressed file with the last 1500 lines from our log. However, we have no idea which line to start from and the size of our source file won’t change.<\/p>\n\n\n\n- head<\/strong> <\/li><\/ul>\n\n\n\n
-n $((TOTAL_LINES – 1500)) BIG_BIG_LOGFILE.log<\/span> will show a standard output of the whole file but for the last 1500 lines. In this case, there is no sense in using this command, since there is no space on the disk to save it; it will take 30-40 minutes to execute such an operation.<\/p>\n\n\n\n- grep<\/strong><\/li><\/ul>\n\n\n\n
-grep -e \u00ab[..\/Oct\/2012:..:..:.. -0200]\u00bb BIG_BIG_LOGFILE.log<\/span> will show a standard output of the whole period of October 2012. The output can be compressed as well as the output of the tail command. Just like in the case of tail, the size of our source file won’t change. Inverse filtering is possible, but the available free space is not enough to storage its output.<\/p>\n\n\n\n- split<\/strong><\/li><\/ul>\n\n\n\n
is very similar to what I decided to do, however, it could split by rows or bytes, but I wanted to have it by months, as if it was rotated according to the schedule. Moreover, the Linux <\/strong>distribution package that contained the problem had a version that couldn’t transfer the standard output to another program (in our case, used for compression), and there wasn’t enough space on the disk for another similar volume.<\/p>\n\n\n\n- awk<\/strong><\/li><\/ul>\n\n\n\n
a full-fledged programming language and almost a runtime environment, specially created for processing text information. Within it, you can implement all the above mentioned functions and much more, but life had other ideas and I used python to do what I intended to do…<\/p>\n\n\n\n
Yes, python<\/strong>; of course, it was possible to use awk, but it was just coincident with the fact that I wanted to try python in a real task, even if it was such a contrived one. Undoubtedly, you can write the task-solving code using C, but python attracted me, like a new shining toy held by a child, by its syntactical sugar, convenient arrays with slices of them made by REPL <\/strong>from iPython<\/strong> and the rest wonderful things that were to be mastered.<\/p>\n\n\n\nWith the four functions at hand: read<\/strong>, seek<\/strong>, write<\/strong>, truncate <\/strong>\u2013 the algorithm looked like this:<\/p>\n\n\n\n- Search for the line in our log file with which the LAST MONTH begins. <\/li>
- Output to a separate compressed file with the month and year in the file name: XXXX-Mon-2012.log.gz <\/li>
- Cutting off the tail of the file starting from this line. Truncate function. <\/li>
- Repeat steps 1, 2, 3 until I am content with the remaining size.<\/li><\/ol>\n\n\n\n
Before that, it was necessary to redirect the log to another file, since apache doesn’t know anything about step 3 and reduction of the file size doesn’t exist for it. The very first its record will return the file size to the original 135 GB<\/strong>, with the cut off space being filled with 0<\/strong> instead of the deleted content, but this won’t reduce the occupied disk space.<\/p>\n\n\n\nFor writing and testing, iPython <\/strong>environment was installed that adds quite a number of convenient goodies to a common python input, like for example, command history and auto-completion of the entered commands and the object methods\/parameters.<\/p>\n\n\n\nTaking impressive size of the file and almost 100% ascending order of all records into consideration, the first step should have been done by the classical binary search and the basic file operations read and seek. But there is one catch – seek function gets the bytes position as a parameter, but we need to get the exact beginning of the line; we need to find not a single but several values \u200b\u200bby the number of months in the file. Therefore, a combined method was applied, with the binary search used to select a sufficiently large area that will definitely contain several records from 3 to 10. In the resulting area, we successively search for a probable month shift.<\/p>\n\n\n\n
To tell the truth, I haven’t written the program to the end; the file was successfully cut into pieces, which were compressed right when being saved on the disk. All the necessary functions have been written and they were called directly from iPython console. Of course, other functions appeared in addition to the below-mentioned , but with those my ideas clearly led me in a wrong direction and I just won\u2019t include them in the blog.<\/p>\n\n\n\n
I began with writing a function that extracted a date from the log line and converted it into a python date; the Regexp module makes it possible to do this in almost 1 line:<\/p>\n\n\n\n
import re\n def date_in_str(line):\n\tgot_it = re.search(\"\\[(?P<date>.*?)(?= ) (?P<timezone>.*?)\\]\",line)\n\tif None != got_it:\t\t\n\t\ttry:\n\t\t\tdt = datetime.datetime.strptime(got_it.group(1),'%d\/%b\/%Y:%H:%M:%S')\n\t\texcept ValueError:\n\t\t\tdt = None\n\treturn dt<\/code><\/pre>\n\n\n\nThen, let’s use this function to search for the first date in an arbitrary area read from our giant file:<\/p>\n\n\n\n
def first_date(in_buf):\n\tline_start = 0\n\tall_lines = in_buf.splitlines()\n\tfor my_line in all_lines:\n\t\tdt = date_in_str(my_line)\n\t\tif None != dt:\t\t\t\n\t\t\treturn line_start,dt,my_line\n\t\tline_start = line_start + len(my_line)<\/code><\/pre>\n\n\n\n<\/p>\n\n\n\n
in_buf<\/span> \u2013 a buffer, an array of symbols read by read function.<\/p>\n\n\n\nThen, a function was written that determined if there was a month change in the rows in the arbitrary buffer and returning an index in the line start buffer if there any:<\/p>\n\n\n\n
def find_month_change(in_buf):\n\tall_lines = in_buf.splitlines()\n\tprev_date = 0\n\tline_start = 0\n\tfor my_line in all_lines:\n\t\tdt = date_in_str(my_line)\n\t\tif None != dt:\n\t\t\tsys.stdout.write('.')\n\t sys.stdout.flush()\n\t\t\tif prev_date == 0:\n\t\t\t\tprev_date = dt\n\t\t\telif prev_date.month != dt.month:\n\t\t\t\treturn line_start\n\t\t\telse:\n\t\t\t\tprev_date = dt\n\t\tline_start = line_start + len(my_line) + 1\n\treturn None<\/code><\/pre>\n\n\n\nAnother small function that returned the first found date and an offset <\/em>in the file from the offset position that read buf_size<\/strong> bytes at a time.<\/p>\n\n\n\nbuf_size = 1024*64\ndef get_date_in_file(my_file,offset):\n my_file.seek(offset,0)\n buf = f_size.read(buf_size)\t\n return first_date(buf)<\/code><\/pre>\n\n\n\nAll we are left to do is to combine them into a function that uses the binary search to find the border line of the last month. Here is the first option:<\/p>\n\n\n\n
def get_it(my_file,start):\n\tfirst_offset,first_date = get_date_in_file(my_file,start)\n\n\tfirst_month = first_date.year * 12 + first_date.month;\n\n\tmy_file.seek(0,2)\n\tf_size = my_file.tell()\n\tposition = f_size - buf_size\n\n\tlast_offset,last_date = get_date_in_file(my_file, position)\n\tlast_month = last_date.year * 12 + last_date.month\n\n\tnot_found = true\n\tend_shr = f_size \n\twhile not_found:\n\t\tposition = start + int((end_shr - start)\/ 2) \u2013 buf_size\n\n\t\toffset,mid_date = get_date_in_file(my_file, position)\t\t\n\t\tmid_month = mid_date.year * 12 + mid_date.month\n\t\tmid_date_offset = position + offset\n\n\t\tif mid_month == ( last_month \u2013 1):\n\t\t month_chng_offset = find_month_change()\n\t\t if month_chng_offset != None:\n\t\t\tnot_found = false\n\t\t\t#return month_chng_offset\n\t\telse if mid_month == last_month:\n\t\t end_shr = mid_date_offset\n\t\telse:\n\t\t start = mid_date_offset<\/code><\/pre>\n\n\n\nThis function is not immune from mistakes; it is written on the hypothesis that:<\/p>\n\n\n\n
- Records are ordered strictly ascending \u2013 this condition is almost 100% fulfilled; the date backlash is insignificant and doesn\u2019t affect the algorithm stability. <\/em><\/li>
- There are records for all months \u2013 in this case, it is 100% fulfilled.<\/em><\/li>
- A month change will definitely fall into the read buffer, i.e. the last buffer line won\u2019t be the last month line or an incomplete line of the next month, from which it\u2019s impossible to read the whole date. \u2013 In this case, as luck would have it.<\/em><\/li><\/ul>\n\n\n\n
Upon second thought, it becomes clear that step 3 sometimes may not be fulfilled. The algorithm for reading lines with dates may be supplemented by checking that the records very close to the end of the last day of the month were read and to start reading the lines one by one until a line with the next day date is encountered.<\/p>\n\n\n\n
Despite the above-mentioned disadvantages, the function fulfilled the task. I called it up manually in iPyhon, using parameters to pass the open file and the offset sufficient to find a month. From time to time, I randomly selected dates from various areas of the file, making an approximate map of the records distribution. If the function failed, I called it up with a different array and it returned the real index of the line start and the month shift.<\/p>\n\n\n\n
After receiving the long hoped-for index, I called up another function, saving all the compressed data starting with this index:<\/p>\n\n\n\n
import subprocess\n\ndef compress_tail(in_file,out_file):\n compressor = subprocess.Popen(['gzip','-','6'],1,None,subprocess.PIPE,out_file,subprocess.PIPE)\n buff = in_file.read(1024*1024)\n while True:\n\tcompressor.stdin.write(buff)\n\tif len(buff) < 1024*1024:\n\t has_data = False;\n\t break\n\tbuff = in_file.read(1024*1024)\n compressor.communicate()\n return compressor<\/code><\/pre>\n\n\n\nKeeping it simple and not connecting gzip modules and something like them, we think of Unix Way and call up a child process with the connected open files STDIN = our giant log file with the fixed offset at the beginning of the month STDOUT = a new open file with the name with \u201cgz\u201d extension containing month and year. We feed the tail of our log to the child process by 1 Mb portions.<\/p>\n\n\n\n
The last step is calling up truncate <\/strong>(offset-1). <\/p>\n\n\n\n<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
A long time ago in a galaxy far far away, when analyzing a remaining free space on a certain Linux server that served about 50-100 websites of diverse…<\/p>\n","protected":false},"author":6,"featured_media":705,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[14],"tags":[],"class_list":["post-684","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-web-servers"],"read_time":6,"_links":{"self":[{"href":"https:\/\/hostry.com\/blog\/wp-json\/wp\/v2\/posts\/684","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hostry.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hostry.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hostry.com\/blog\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/hostry.com\/blog\/wp-json\/wp\/v2\/comments?post=684"}],"version-history":[{"count":19,"href":"https:\/\/hostry.com\/blog\/wp-json\/wp\/v2\/posts\/684\/revisions"}],"predecessor-version":[{"id":713,"href":"https:\/\/hostry.com\/blog\/wp-json\/wp\/v2\/posts\/684\/revisions\/713"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/hostry.com\/blog\/wp-json\/wp\/v2\/media\/705"}],"wp:attachment":[{"href":"https:\/\/hostry.com\/blog\/wp-json\/wp\/v2\/media?parent=684"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hostry.com\/blog\/wp-json\/wp\/v2\/categories?post=684"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hostry.com\/blog\/wp-json\/wp\/v2\/tags?post=684"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}