Archiving

ZettaLogs log management service, has a finite retention period depending on the plan. After that period the logs are removed from the system. Therefore, if you do not store your logs anywhere else you may want to consider the log archiving option. ZettaLogs offers archiving of logs to Amazon AWS S3 service in raw and/or structured json format.

ZettaLogs supports log archiving to customer AWS S3 bucket as well as managed archiving. While AWS S3 archiving of logs comes with no extra cost with paid plans, managed log archiving is an extra feature that must be selected with a plan and incurs extra costs.

For AWS S3 log archiving, ZettaLogs do not create or manage AWS account or S3 bucket for you. You have to create and manage your AWS account and S3 bucket. After you create an AWS account, you need to create an S3 bucket. Log in to your AWS account and create an S3 bucket. After creating the bucket, select it from the All Buckets panel. Then you need to set permissions of the bucket:

  • Click the Properties button
  • Select the Permissions tab
  • Click Add more permissions
  • In the Grantee box of the new line that appears, input the canonical Id of ZettaLogs:
    6d823f8a60c8136628a65c14943c3eef8a47ecbbb44330d3712365193a3e9ae4
  • Select the check boxes next to List, Upload/Delete, View Permissions.

AWS S3 Bucket Permissions

Now head back to ZettaLogs Project Settings page and select Archiving tab. All you need to do is to input the name of the S3 bucket into the text box and save it. As soon as the configuration is saved, ZettaLogs starts uploading gzipped archive files, hourly. Each archive file may consist of a raw format file, a structured json format file or both, depending on the choice. The status of the last backup run may be observed from the color of the circle right next to the bucket name. If that color is green then everything went OK. However, if it is red then it means that something was wrong with the last backup. The status and date of the last backup may be observed by hovering over the status icon.

Log Archiving Options

Timestamping

In addition, time-stamping service may also be opted. In that case, a time-stamp file will be generated for each archive file. The archive file and the time-stamp file will be combined into a single tar file and uploaded to the S3 bucket. Time-stamping ensures that the archive file was created prior to the date included in the time-stamp file. This type of archiving may be beneficial for security compliance purposes. Time-stamping procedure and time-stamp file complies with RFC3161.

Free time-stamping is available via ZettaLogs’ own time-stamping service or Free TSA service. In addition, paid time-stamping services can be made available by selecting “certified time-stamping” during plan subscription. If your subscription includes certified time-stamping, then you may select one of the paid trusted time-stamping services with a trusted root certificate.

Timestamp Verification

The time-stamp of the archive file may be verified using the OpenSSL tool. Assume that we have downloaded a time-stamped archive file with just a raw ormat from S3.


$ tar -xf 9251e007d8484425942b0d6d5752fb01-2016-05-24-07.27.raw-timestamped.tar
$ openssl ts -verify -data 9251e007d8484425942b0d6d5752fb01-2016-05-24-07.27.raw.gz -in 9251e007d8484425942b0d6d5752fb01-2016-05-24-07.27.raw.gz.tsr -CAfile freetsa_cacert.pem
Verification: OK

Managed Archiving

Managed archiving is an extra feature that must be selected during plan subscription. In this case ZettaLogs stores and manages archive files for you. To activate the feature you must check the “Managed Archiving” box in “Archive” tab in “Project Settings”. In this mode, even though you do not need to supply an AWS S3 bucket name, you may supply it for archiving to both locations.

Managed Log Archiving Options

For managed archiving users, a new tab named “Archive” appears on the top toolbar. You may view and manage your archive files from this tab.