Difference between revisions of "Email Statistics"
From SME Server
Jump to navigationJump to search| Line 2: | Line 2: | ||
==Built-In Email Reports== | ==Built-In Email Reports== | ||
| − | + | Some email statistics can be seen from server-manager under "Administration", "[[Mail_log_file_analysis|Mail log file analysis]]" | |
| − | Some email statistics can be seen from server-manager under "Administration", "Mail log file analysis" | ||
| − | |||
| − | |||
| − | |||
| + | === qpsmtpd: Incoming SMTP traffic === | ||
SME 7.2 and later include the 'logterse' plugin to qpsmtpd as well as the 'qplogsumm.pl' statistics script. | SME 7.2 and later include the 'logterse' plugin to qpsmtpd as well as the 'qplogsumm.pl' statistics script. | ||
| − | + | All in-bound SMTP connections are handled by qpsmtpd. In addition to the qplogsumm ''summary'' information described here, you can view the raw qpsmtpd logs as described in [[Mail_log_file_analysis]]. | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | ===qplogsumm.pl=== | |
qplogsumm.pl updates /var/log/qpsmtpd/state with per-plugin statistics for any qpsmtpd plugin that appears in a qpsmtpd logterse entry each time the qpsmtpd log is rotated. | qplogsumm.pl updates /var/log/qpsmtpd/state with per-plugin statistics for any qpsmtpd plugin that appears in a qpsmtpd logterse entry each time the qpsmtpd log is rotated. | ||
| Line 34: | Line 16: | ||
e-smith-viewlogfiles-1.8.0-4 (released Nov 28 2007) or later will allow you to 'View' /var/log/qpsmtpd/state from 'View log files' in the server-manager (earlier versions conceal all files named "state" - [[bugzilla:3416|Bug 3416]]). | e-smith-viewlogfiles-1.8.0-4 (released Nov 28 2007) or later will allow you to 'View' /var/log/qpsmtpd/state from 'View log files' in the server-manager (earlier versions conceal all files named "state" - [[bugzilla:3416|Bug 3416]]). | ||
| − | + | ====Enable qplogsumm==== | |
qplogsumm.pl is disabled by default in smeserver-qpsmtpd-1.2.1-52.el4.sme and later ([[bugzilla:3727|Bug 3727]]). Enable it with | qplogsumm.pl is disabled by default in smeserver-qpsmtpd-1.2.1-52.el4.sme and later ([[bugzilla:3727|Bug 3727]]). Enable it with | ||
config setprop qpsmtpd qplogsumm enabled | config setprop qpsmtpd qplogsumm enabled | ||
signal-event email-update | signal-event email-update | ||
| − | + | ====Force first log rotation==== | |
qplogsumm.pl only updates its statistics when the qpsmtpd log file is rotated. This can take several days on a moderately busy server, and could take weeks or months on some servers. | qplogsumm.pl only updates its statistics when the qpsmtpd log file is rotated. This can take several days on a moderately busy server, and could take weeks or months on some servers. | ||
| Line 45: | Line 27: | ||
kill -ALRM `cat /var/service/qpsmtpd/log/supervise/pid` | kill -ALRM `cat /var/service/qpsmtpd/log/supervise/pid` | ||
| − | + | ====Potential Problems==== | |
| − | + | =====/var/log/qpsmtpd/state missing===== | |
qplogsumm.pl will completely lock all in-bound email if <tt>/var/log/qpsmtpd/state</tt> is missing when multilog attempts to rotate the qpsmtpd log file ([[bugzilla:3393|Bug 3393]]). This will never happen under normal circumstances - only in the event of a disk error or if the administrator moves or deletes the existing file. If this does happen, the problem can be resolved using | qplogsumm.pl will completely lock all in-bound email if <tt>/var/log/qpsmtpd/state</tt> is missing when multilog attempts to rotate the qpsmtpd log file ([[bugzilla:3393|Bug 3393]]). This will never happen under normal circumstances - only in the event of a disk error or if the administrator moves or deletes the existing file. If this does happen, the problem can be resolved using | ||
touch /var/log/qpsmtpd/state | touch /var/log/qpsmtpd/state | ||
sv restart /var/log/qpsmtpd | sv restart /var/log/qpsmtpd | ||
| − | + | =====Unprocessed Log Files===== | |
If qpsmtpd is terminated abnormally (due to a power failure, for example), the log files may not be completely "processed". If this happens, you will have files in <tt>/var/log/qpsmtpd</tt> with names like the one shown below, ending in '''.u''':<code> @4000000048ec03873b1a841c.u</code> | If qpsmtpd is terminated abnormally (due to a power failure, for example), the log files may not be completely "processed". If this happens, you will have files in <tt>/var/log/qpsmtpd</tt> with names like the one shown below, ending in '''.u''':<code> @4000000048ec03873b1a841c.u</code> | ||
The transactions in these '''.u''' log files will '''not''' be included in the summary information in <tt>/var/log/qpsmtpd/state</tt> | The transactions in these '''.u''' log files will '''not''' be included in the summary information in <tt>/var/log/qpsmtpd/state</tt> | ||
| − | + | === qmail: Outgoing SMTP traffic === | |
| − | + | ''qmail'' log file analysis and some statistics are described in [[Mail_log_file_analysis]] | |
| − | + | ||
| − | + | Note that since all spam filtering is done by qpsmtpd, ''qmail'' log files or analysis tools will contain spam filtering statistics. | |
| − | + | ==Contribs & Addons== | |
| − | + | ===Brian Read's spamfilter-stats-7.pl=== | |
| − | |||
Brian Read's mailstats contrib analyzes your qpsmtpd log files and sends an email to the specified email address summarizing your SME server activity. | Brian Read's mailstats contrib analyzes your qpsmtpd log files and sends an email to the specified email address summarizing your SME server activity. | ||
Full details can be found at '''[[mailstats]]''' | Full details can be found at '''[[mailstats]]''' | ||
| − | + | ===Qmail_Statistics_(AWStats)=== | |
Michael Weinberger has assembled a script that allows you to easily install awstats and configure it to provide email delivery statistics. | Michael Weinberger has assembled a script that allows you to easily install awstats and configure it to provide email delivery statistics. | ||
Full details can be found at '''[[Qmail_Statistics_(AWStats)]]''' | Full details can be found at '''[[Qmail_Statistics_(AWStats)]]''' | ||
| − | + | ===qplogtail=== | |
qplogtail is a script intended to help monitor /var/log/qpsmtpd/current and extract a concise but meaningful display of what the server is up to. | qplogtail is a script intended to help monitor /var/log/qpsmtpd/current and extract a concise but meaningful display of what the server is up to. | ||
| Line 122: | Line 103: | ||
Direct comments or questions to [[Bugzilla:3418]] | Direct comments or questions to [[Bugzilla:3418]] | ||
| − | + | ===qploggrep=== | |
<b>qploggrep</b> allows you to search your existing qpsmtpd logs as though they had been generated by <b>qplogtail</b>, then display matching results. | <b>qploggrep</b> allows you to search your existing qpsmtpd logs as though they had been generated by <b>qplogtail</b>, then display matching results. | ||
| Line 148: | Line 129: | ||
* ''qploggrep'' uses a case-insensitive search, so <tt>qploggrep abc</tt> will locate lines containing ''abc'', ''ABC'', ''aBc'', etc. | * ''qploggrep'' uses a case-insensitive search, so <tt>qploggrep abc</tt> will locate lines containing ''abc'', ''ABC'', ''aBc'', etc. | ||
| − | + | ===[[Qpsmtpd_connection_time]]=== | |
See [[qpsmtpd_connection_time]] | See [[qpsmtpd_connection_time]] | ||
---- | ---- | ||
[[Category:Howto]] | [[Category:Howto]] | ||
| + | [[Category:Administration]] | ||
| + | [[Category:Mail]] | ||
Revision as of 18:14, 8 November 2008
Various options for monitoring your mail server performance
Built-In Email Reports
Some email statistics can be seen from server-manager under "Administration", "Mail log file analysis"
qpsmtpd: Incoming SMTP traffic
SME 7.2 and later include the 'logterse' plugin to qpsmtpd as well as the 'qplogsumm.pl' statistics script.
All in-bound SMTP connections are handled by qpsmtpd. In addition to the qplogsumm summary information described here, you can view the raw qpsmtpd logs as described in Mail_log_file_analysis.
qplogsumm.pl
qplogsumm.pl updates /var/log/qpsmtpd/state with per-plugin statistics for any qpsmtpd plugin that appears in a qpsmtpd logterse entry each time the qpsmtpd log is rotated.
Sample output here
e-smith-viewlogfiles-1.8.0-4 (released Nov 28 2007) or later will allow you to 'View' /var/log/qpsmtpd/state from 'View log files' in the server-manager (earlier versions conceal all files named "state" - Bug 3416).
Enable qplogsumm
qplogsumm.pl is disabled by default in smeserver-qpsmtpd-1.2.1-52.el4.sme and later (Bug 3727). Enable it with
config setprop qpsmtpd qplogsumm enabled signal-event email-update
Force first log rotation
qplogsumm.pl only updates its statistics when the qpsmtpd log file is rotated. This can take several days on a moderately busy server, and could take weeks or months on some servers.
You can force a rotation of the qpsmtpd log files in order to generate initial data in /var/log/qpsmtpd/state using
kill -ALRM `cat /var/service/qpsmtpd/log/supervise/pid`
Potential Problems
/var/log/qpsmtpd/state missing
qplogsumm.pl will completely lock all in-bound email if /var/log/qpsmtpd/state is missing when multilog attempts to rotate the qpsmtpd log file (Bug 3393). This will never happen under normal circumstances - only in the event of a disk error or if the administrator moves or deletes the existing file. If this does happen, the problem can be resolved using
touch /var/log/qpsmtpd/state sv restart /var/log/qpsmtpd
Unprocessed Log Files
If qpsmtpd is terminated abnormally (due to a power failure, for example), the log files may not be completely "processed". If this happens, you will have files in /var/log/qpsmtpd with names like the one shown below, ending in .u: @4000000048ec03873b1a841c.u
The transactions in these .u log files will not be included in the summary information in /var/log/qpsmtpd/state
qmail: Outgoing SMTP traffic
qmail log file analysis and some statistics are described in Mail_log_file_analysis
Note that since all spam filtering is done by qpsmtpd, qmail log files or analysis tools will contain spam filtering statistics.
Contribs & Addons
Brian Read's spamfilter-stats-7.pl
Brian Read's mailstats contrib analyzes your qpsmtpd log files and sends an email to the specified email address summarizing your SME server activity.
Full details can be found at mailstats
Qmail_Statistics_(AWStats)
Michael Weinberger has assembled a script that allows you to easily install awstats and configure it to provide email delivery statistics.
Full details can be found at Qmail_Statistics_(AWStats)
qplogtail
qplogtail is a script intended to help monitor /var/log/qpsmtpd/current and extract a concise but meaningful display of what the server is up to.
qplogtail extracts 6 kinds of information:
- Normal connections:
28545 Accepted connection 4/30 from 86.139.2.73 ... - Errors in violation of Instances:
5146 Too many connections: 40 >= 40. Waiting one second. - Errors in violation of InstancesPerIP:
5320 hosts_allow plugin: Too many connections from 212.100.229.201: 6 > 5Denying connection. - Messages blocked by any qpsmtpd plugin:
15751 logging::logterse plugin: ` 82.210.181.241 241-pra-6.acn.waw.pl 241-pra-6.acn.waw.pl <Glasteinzhza@ask-it-here.com> dnsbl 903 http://www.spamhaus.org/query/bl?ip=82.210.181.241 msg denied before queued - Messages queued for delivery:
15587 logging::logterse plugin: ` 128.220.32.40 miami.deuvis.com miami.deuvis.com <aapple@deuvis.com> <c.wolf@ncxr.org> queued <200709270344.l8R3iq0b010299@deuvis.com> No, hits=-2.6 required=5.0_ - Connection time values from the connection_time plugin (if present)
@4000000048641d5c0951f6a4 15110 connection_time plugin: Connection time from 209.74.246.66: 1.566 sec.
Each normal smtp transaction will generate two lines of output containing:
msgid remote_ip x/40 msgid remote_ip dispostion details
If you have the connection_time plugin installed, you will also get:
msgid remote_ip timeconnected
Sample output:
# qplogtail 14868 209.74.246.66 0/40 14868 209.74.246.66 check_basicheaders msg denied before queued 14868 209.74.246.66 1.622 sec. 14879 200.127.59.114 0/40 14879 200.127.59.114 dnsbl msg denied before queued 14879 200.127.59.114 2.874 sec. 14890 69.147.64.214 0/40 14890 69.147.64.214 queued No, hits=-2.6 required=5.0_ 14890 69.147.64.214 7.433 sec.
To install:
cd /usr/local/bin wget -O qplogtail http://bugs.contribs.org/attachment.cgi?id=2035 chmod 755 qplogtail
To run:
qplogtail
Direct comments or questions to Bugzilla:3418
qploggrep
qploggrep allows you to search your existing qpsmtpd logs as though they had been generated by qplogtail, then display matching results.
To install:
cd /usr/local/bin wget -O qploggrep http://bugs.contribs.org/attachment.cgi?id=2034 chmod 755 qploggrep
To Run:
- Search all existing qpsmtpd logs for email to or from user@domain.tld:
qploggrep user@domain.tld
- Search for email to or from user@domain.tld that was denied by spamassassin:
qploggrep spamassassin | grep user@domain.tld
- Display all qpsmtpd transactions denied due to dnsbl:
qploggrep dnsbl
- Display the total connection time for all connections, sorted by connection time (assumes that you have installed and enabled the connection_time plugin):
qploggrep connection_time | sort -k 3 -n
- Display all info from /var/log/qpsmtpd/* (note the space and dot)
qploggrep .
- Show all lines recording "connection x of y", sorted by the number of concurrent connections
qploggrep "/`config getprop smtpd Instances` " | sort -k4
Program Notes:
- qploggrep cannot locate information that is not there. For example, since the dnsbl plugin drops the incoming connection before the remote server specifies the addressee, you cannot find any addressee information for messages blocked by dnsbl.
- qploggrep uses a case-insensitive search, so qploggrep abc will locate lines containing abc, ABC, aBc, etc.
