Sending Logs

ZettaLogs accepts log lines over either clear TCP or SSL/TLS connection. Each log line is expected to be structured according to RFC5424. The preferred way of sending logs is to use a syslog daemon. For most linux/unix systems, either rsyslog or syslog-ng can be used for this purpose. Rsyslog is the default syslog daemon for Debian/Ubuntu, Fedora and openSUSE.

Project Token

Before configuring your system to send logs to ZettaLogs, you must identify your Project Token. The project token is the identifier that you must enter into your syslog configuration so that ZettaLogs knows which project the received log event belongs to. Anyone who has the knowledge of the project token can send log events into your project. The project token is your project’s secret key so that only you and no one else may send log events into your project. You may find the token in your project’s settings view.

Project Token

Rsyslog

Rsyslog can be configured to send log lines to ZettaLogs by adding configuration files into its configuration directory which is “/etc/rsyslog.d”. Configuration file format differs according to its version. Version 6.x and below uses a different format than version 7.x and above. Although version 7.x accepts old format it is recommended that new format is used since newer versions may drop the support for old format.

The version information of the rsyslog daemon can be obtained by issuing the following command on the command prompt.

$ rsyslogd -v

We recommend updating Rsyslog to the latest 8.x version by following the instructions for Ubuntu or Debian.

Clear TCP

Clear TCP is not recommended. One of the reasons is that the log lines are transferred openly without any encryption over the internet. In addition to that and more importantly, the client (rsyslogd) cannot verify the authenticity of the server (ZettaLogs) thus rendering the communication vulnerable to man-in-the-middle attacks.

Create a file /etc/rsyslog.d/25-zetta.conf with the following content for rsyslog version 6.x and below.


$WorkDirectory /var/spool/rsyslog # default location for work (spool) files
$ActionQueueType LinkedList       # use asynchronous processing
$ActionQueueFileName zettalogs    # set file name, also enables disk mode
$ActionResumeRetryCount -1        # infinite retries on insert failure
$ActionQueueSaveOnShutdown on     # save in-memory data if rsyslog shuts down
$ActionQueueMaxDiskSpace 1g       # maximum size that all queue files together will use on disk

# Define ZettaLogs log line format which is according to RFC5424
$template ZettaFormat,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid% [PROJECT-TOKEN@46644] %msg%\n"

# Send logs
*.* @@data.zettalogs.com:514;ZettaFormat

You must insert your 32 character project token into the place marked as PROJECT-TOKEN.

For rsyslog version 7.x and above the following configuration file may be used.


$WorkDirectory /var/spool/rsyslog # default location for work (spool) files
$ActionQueueType LinkedList       # use asynchronous processing
$ActionQueueFileName zettalogs    # set file name, also enables disk mode
$ActionResumeRetryCount -1        # infinite retries on insert failure
$ActionQueueSaveOnShutdown on     # save in-memory data if rsyslog shuts down
$ActionQueueMaxDiskSpace 1g       # maximum size that all queue files together will use on disk

# Define ZettaLogs log line format which follows RFC5424 standard
template(name="ZettaFormat" type="string" string="<%pri%>%protocol-version% %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid% [PROJECT-KEY@46644] %msg%\n")

# Send logs
action(type="omfwd" protocol="tcp" target="data.zettalogs.com" port="514" template="ZettaFormat")

SSL/TLS

Sending logs over SSL/TLS connection is the safe and recommended method. rsyslog-gnutls package must be installed on the system. You may install this package on Debian/Ubuntu systems by issuing the following command.

$ sudo apt-get install rsyslog-gnutls

Create a file /etc/rsyslog.d/25-zetta.conf with the following content for rsyslog version 6.x and below.


$WorkDirectory /var/spool/rsyslog # default location for work (spool) files
$ActionQueueType LinkedList       # use asynchronous processing
$ActionQueueFileName zettalogs    # set file name, also enables disk mode
$ActionResumeRetryCount -1        # infinite retries on insert failure
$ActionQueueSaveOnShutdown on     # save in-memory data if rsyslog shuts down
$ActionQueueMaxDiskSpace 1g       # maximum size that all queue files together will use on disk

# certificate files - just CA for a client
$DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-certificates.crt
# Use this only if the root certificate is not installed on the system
#$DefaultNetstreamDriverCAFile /etc/rsyslog.d/keys/zettalogs.com-bundle.crt

# set up the action
$ActionSendstreamDriver gtls      # use gtls netstream driver
$ActionSendStreamDriverMode 1     # require TLS for the connection
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer *.zettalogs.com

# Define ZettaLogs log line format which is according to RFC5424
$template ZettaFormat,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid% [PROJECT-TOKEN@46644] %msg%\n"

# Send logs
*.* @@data.zettalogs.com:6514;ZettaFormat

You must insert your 32 character project token into the place marked as PROJECT-TOKEN.

For rsyslog version 7.x and above the following configuration file may be used.


$WorkDirectory /var/spool/rsyslog # default location for work (spool) files
$ActionQueueType LinkedList       # use asynchronous processing
$ActionQueueFileName zettalogs    # set file name, also enables disk mode
$ActionResumeRetryCount -1        # infinite retries on insert failure
$ActionQueueSaveOnShutdown on     # save in-memory data if rsyslog shuts down
$ActionQueueMaxDiskSpace 1g       # maximum size that all queue files together will use on disk

# certificate files - just CA for a client
$DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-certificates.crt
# Use this only if the root certificate is not installed on the system
#$DefaultNetstreamDriverCAFile /etc/rsyslog.d/keys/zettalogs.com-bundle.crt

# Define ZettaLogs log line format which follows RFC5424 standard
template(name="ZettaFormat" type="string" string="<%pri%>%protocol-version% %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid% [PROJECT-TOKEN@46644] %msg%\n")

# Send logs
action(type="omfwd" protocol="tcp" target="data.zettalogs.com" port="6514" template="ZettaFormat" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" StreamDriverPermittedPeers="*.zettalogs.com")

The default maximum supported message size by Rsyslog is 8k. We recommend increasing this to 64k by adding the following line to the top of your /etc/rsyslog.conf file.

$MaxMessageSize 64k

SSL/TLS configuration needs trusted root certificates to be installed on the system. ZettaLogs certificates are issued by StartCom Ltd. certificate authority. Normally the root certificate of StartCom should be installed on your system by default. If for some reason it is NOT installed then the communication to our servers won’t be authenticated by rsyslogd and the logs won’t be sent. In that case, please download the bundled certificate from our server and install it as described below.


$ sudo mkdir -pv /etc/rsyslog.d/keys
$ cd /etc/rsyslog.d/keys
$ sudo wget https://zettalogs.com/files/zettalogs.com-bundle.crt

Or if wget is not installed you may also use curl instead of the last line above:

$ sudo curl -O https://zettalogs.com/files/zettalogs.com-bundle.crt

Uncomment the line in configuration file starting with “#$DefaultNetstreamDriverCAFile”. And do not forget to restart the rsyslog daemon:

$ sudo service rsyslog restart

Following Files

Rsyslog may be configured to follow any file on the system. As log lines are added into the file being followed, rsyslog will read them and put into its queue. We will show this configuration over an example which will setup nginx access and error log files for sending to ZettaLogs.

Create a file /etc/rsyslog.d/24-zetta-files.conf with the following content.


$ModLoad imfile
$InputFilePollInterval 10 
$PrivDropToGroup adm
$WorkDirectory /var/spool/rsyslog


## File example.com_access.log
$InputFileName /var/log/nginx/example.com_access.log
$InputFileTag example.com_access            # set an identifying tag
$InputFileStateFile stat-example.com_access # state file saved under /var/spool/rsyslog (must be a unique file name)
$InputFileSeverity info                     # Set the severity value
#$InputFileFacility facility                # Set the facility value if needed. It defaults to local0 (16)
$InputFilePersistStateInterval 10000
$InputRunFileMonitor
#############################################

## File example.com_error.log
$InputFileName /var/log/nginx/example.com_error.log
$InputFileTag example.com_error            # set an identifying tag
$InputFileStateFile stat-example.com_error # state file saved under /var/spool/rsyslog (must be a unique file name)
$InputFileSeverity info                    # Set the severity value
#$InputFileFacility facility               # Set the facility value if needed. It defaults to local0 (16)
$InputFilePersistStateInterval 10000
$InputRunFileMonitor
#############################################

Note that the $InputFileTag will show as app_name field in ZettaLogs. Therefore, set this to the name of the file or to any other unique identifier that makes you identify the source of this log easily.

You may insert as many file monitor blocks (starting with $InputFileName, ending with $InputRunFileMonitor) as you may wish to follow all the files you want to be followed.

With the above setting in place, rsyslog will now follow the files we want. However, it will also output the log lines read from these files to /var/log/syslog. We do not want this because they are already saved in their own log files. Thus add the following filtering configuration to /etc/rsyslog.d/25-zetta.conf to discard the log line from the files we follow right after they are forwarded to ZettaLogs.

...

# Send logs
*.* @@data.zettalogs.com:6514;ZettaFormat
# Discard logs read from files after sending them to ZettaLogs
if $programname == 'example.com_access' then stop
if $programname == 'example.com_error' then stop
# Regular expressions may also be used to match the programname
# if re_match($programname, 'example\\.com.*') then stop

Do not forget to restart the rsyslog daemon after configuration changes.

$ sudo service rsyslog restart