Log Analyzer Commands
The asadm command has a built-in interactive shell for analysis of Aerospike server logs.
Starting the log analyzer
To start the log analyzer shell, run the following command:
asadm -l -f <asLogpath-1> <asLogpath-2> <asLogpath-3> ...
where <asLogpath*> are the space-separated paths of single log files or directories of log files.
Per-session lists of log files
In each session, the log analyzer keeps lists of log files for analysis:
- All list: the complete list of all log files added to the session.
- Selected list: those log files currently active for analysis.
In the log analyzer shell itself, you can add, select, or remove logs.
Summary of log analyzer commands
The log analyzer has the following commands. Some of them are similar to common UNIX shell commands.
| Log analyzer command | Description |
|---|---|
add | Add more log files to the All and Selected lists. |
count | Display number of lines that match a specified pattern and time range. |
diff | Display counts of a specified search pattern and the deltas of that pattern in consectuive log lines. |
grep | Display lines that match a specified pattern and time range. |
help | Display usage information about a specified command. |
histogram | Calculate and display histogram measurements for a specific time range. |
list | Display names of logs in the All and Selected lists. |
pager | pager on: Sets paging size of output before pausing. Paging size is specified with the -p option. pager off: Turn off pausing at paging size. |
remove | Remove logs from the All and Selected lists. |
select | Select log files from the All list to put in the Selected lists. |
Command help
The help command displays usage information about the specified command.
Log-analyzer> help grep
Displays all lines from server logs matched with input strings.
Usage: grep
-s - Space-separated strings to search in log files. Format: -s
'string1' 'string2' ... 'stringn'
-a - Set 'AND'ing of search strings (provided with -s): Finding
lines with all search strings in it. Default is 'OR'ing:
Finding lines with at least one search string in it.
-v - Non-matching strings (space separated).
-i - Perform case-insensitive matching of search strings (-s) and
non-matching strings (-v)
-u - Set to find unique lines.
-f - Log time from which to analyze. May use the following formats:
'Sep 22 2011 22:40:14', -3600, or '-1:00:00'
Default: head
-d - Maximum time period to analyze. May use the following formats:
3600 or 1:00:00.
-n - Comma-separated node numbers. You can get these numbers by the
list command. Example: -n '1,2,5'. If not set, then runs on all
server logs in the selected list.
-p - Showing output in pages with p entries per page
Default: 10
About these examples
For details about command options in these examples, use help <command_name>.
Omission of lengthy output lines in some examples is indicated by three periods:
Line 1
.
.
.
Line n
add
The add command makes new logs from different nodes of the cluster available for analysis.
It adds logs to both the All to the Selected lists.
Note: You can add only a single log file per node.
Arguments to add are the space-separated paths of a log or a directory of logs.
Note for Aerospike version 3.7.1 and earlier: the displayed name of a log file is the MD5 hash of the specified path.
In the following example, a directory of logs and a single log are added.
Log-analyzer> add /home/vagrant/shared/data/testlogs/ /var/log/aerospike/aerospike2.log
INFO: Added Log File /home/vagrant/shared/data/testlogs/files/1.log.
INFO: Added Log File /home/vagrant/shared/data/testlogs/files/2.log.
INFO: Added Log File /home/vagrant/shared/data/testlogs/files/3.log.
INFO: Added Log File /home/vagrant/shared/data/testlogs/files/4.log.
INFO: Added Log File /home/vagrant/shared/data/testlogs/files/5.log.
INFO: Added Log File /home/vagrant/shared/data/testlogs/files/6.log.
6 server logs added for server analysis.
INFO: Added Log File /var/log/aerospike/aerospike2.log.
1 server log added for server analysis.
count
The count command displays the number of lines that match a specified pattern and time range.
You can specify any of the following:
- Multiple match patterns.
- Log selection.
- Ignore case.
- Invert match.
- Filter lines that are identical so only a single line is displayed, similar to the Unix
uniqcommand. - Time range.
- Use the
-poption to set the paging number of lines to display, with default 10 lines.
Log-analyzer> count -s "reads" -n "5,6" -f "Jan 31 2017 07:00:00" -d 100 -t 20
~~~~~~~~~~~~~~~~~~~~~~~~~cluster (Page-1)~~~~~~~~~~~~~~~~~~~~~~~~
NODE : c1d635c2a270008 | c81635c2a270008 |
Jan 31 2017 07:00:00: 2 | 2 |
Jan 31 2017 07:00:20: 2 | 2 |
Jan 31 2017 07:00:40: 2 | 2 |
Jan 31 2017 07:01:00: 2 | 2 |
Jan 31 2017 07:01:20: 4 | 4 |
total : 12 | 12 |
diff
The diff command displays the deltas of a specified pattern in consecutive values for search.
It supports following five patterns of search string S and values Vi :
- S V0
- S (V0)
- S (V0, V1, …, Vn)
- S (V0
- V0(V1) S
You can specify any of the following:
- Log selection.
- Ignore case.
- Output filter.
- Filter lines that are identical so only a single line is displayed, similar to the Unix
uniqcommand. - Time range.
- The
-poption to set the paging number of lines to display, with default 10 lines.
Log-analyzer> diff -s "fds - proto" -n "5,6" -f "Jan 31 2017 07:00:00" -d 100
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~fds - proto Diff (Page-1)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE : c1d635c2a270008 . | c81635c2a270008 . |
. : Total Diff | Total Diff |
Jan 31 2017 07:00:00: [0, 510, 510] [0, 510, 510] | [0, 508, 508] [0, 508, 508] |
Jan 31 2017 07:00:10: [0, 510, 510] [0, 0, 0] | [0, 508, 508] [0, 0, 0] |
Jan 31 2017 07:00:20: [0, 510, 510] [0, 0, 0] | [0, 508, 508] [0, 0, 0] |
Jan 31 2017 07:00:30: [0, 510, 510] [0, 0, 0] | [0, 508, 508] [0, 0, 0]
.
.
.
grep
The grep command displays lines that match a specified pattern. You can merge lines from a different file based on timestamp and display lines in ascending order of the timestamp.
You can specify any of the following:
- Multiple match patterns.
- Log selection.
- Ignore case.
- Invert match.
- Filter lines that are identical so only a single line is displayed, similar to the Unix
uniqcommand. - Time range.
- The
-poption to set the paging number of lines to display, with default 10 lines.
Log-analyzer> grep -s "reads" -n "5,6" -f "Jan 31 2017 07:00:00" -d 100
c81635c2a270008::Jan 31 2017 07:00:05 GMT: INFO (info): (hist.c:137) histogram dump: reads (0 total) msec
c1d635c2a270008::Jan 31 2017 07:00:09 GMT: INFO (info): (hist.c:137) histogram dump: reads (0 total) msec
c81635c2a270008::Jan 31 2017 07:00:15 GMT: INFO (info): (hist.c:137) histogram dump: reads (0 total) msec
.
.
.
Histogram
The histogram command analyzes server logs and displays histogram measurements. For background information and types of histograms, see Histograms.
The histogram command parses histogram lines for a specified time range and calculates the percentages of average and maximum operations in each time range.
Note: By default, histogram displays only three buckets of latency percentages. For an example of displaying more buckets with finer granularity, see Increasing buckets and granularity of measurement.
For Aerospike versions 3.9 and later, you can use the -N <namespace> option for a latency histogram for a specific namespace.
Use the -p option to set the paging number of lines to display (default 10 lines).
Log-analyzer> histogram -h reads -f head -d 150 -p 20 -f 'Jan 27 2016 22:26:00'
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~reads Latency (Page-1)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE : node0 . . . | node1 . . . | node2 . . . |
. : % >1ms % >8ms % >64ms ops/sec | % >1ms % >8ms % >64ms ops/sec | % >1ms % >8ms % >64ms ops/sec |
Jan 27 2016 22:26:10: 9.09 6.06 0.00 3.3 | [] [] [] [] | [] [] [] [] |
Jan 27 2016 22:26:20: 0.00 0.00 0.00 0.8 | 3.70 0.00 0.00 2.7 | 11.11 0.00 0.00 0.9 |
Jan 27 2016 22:26:30: 0.00 0.00 0.00 0.0 | 0.00 0.00 0.00 0.1 | 0.00 0.00 0.00 0.0 |
.
.
.
avg : 0.61 0.40 0.00 1.0 | 0.42 0.00 0.00 1.0 | 0.89 0.00 0.00 1.0 |
max : 9.09 6.06 0.00 19.8 | 3.70 0.00 0.00 19.0 | 11.11 0.00 0.00 14.5 |
Syntax variations for histograms for a specific namespace
This same output can be displayed with either of the following command lines:
- Namespace prefix on histogram type. The braces are required:
histogram -h {<namespace>}-write - Namespace with
-N:histogram -h write -N <namespace>
Increasing buckets and granularity of measurement
In a "stretch cluster" configuration, where nodes are geographically distant from each other, you might need percentage of latencies larger than histogram's default display.
Example
You can use a combination of pager on, -b, and -e to more sharply isolate where the latency occurs.
So you can avoid wrapped lines and scroll the display horizontally and vertically, turn on paging:
pager onUse a combination of
-band-eto specify the granulaity of percentages and number of buckets. For example, a combination of-b 4 -e 2displays four buckets of latency in milliseconds in increasing 1, 4, 16, and 64 ratios. The output shows these buckets:% >1ms % >4ms % >16ms % >64ms.Scroll vertically with the arrow keys to see all the buckets and horizontally to see all the values.
Turn off pager mode:
pager off
list
The list command displays the names of server logs currently added and selected for processing.
Log-analyzer> list
Added Logs:
1 : bb9635c2a270008 /var/log/aerospike/aerospike.log
Selected Logs:
bb9635c2a270008 /var/log/aerospike/aerospike.log
pager
The pager command sets paging size of output before pausing or turns the paging off.
pager on: Sets paging size of output before pausing. Paging size is specified with the-poption. When pager mode is on, you can also scroll the output display vertically or horizontally.The command
pager onis especially useful when looking at more than the log analyzer's default number of histogram buckets. For an example, see histogram.pager off: Turn off pausing at paging size.
remove
The remove command removes logs from the All and Selected lists.
To remove all server logs, specify the all argument.
Log-analyzer>
Log-analyzer> remove 1 3 7
INFO: Removed Log File /home/vagrant/shared/data/testlogs/files/6.log.
INFO: Removed Log File /home/vagrant/shared/data/testlogs/files/3.log.
INFO: Removed Log File /var/log/aerospike/aerospike.log.
Log-analyzer>
select
The select command provides a way to select server log from all list to selected list.
It removes existing logs from the All list and adds newly selected logs.
Log-analyzer> list
Added Logs:
1 : MD5_0b32a78b084 /home/vagrant/shared/data/testlogs/files/6.log
2 : MD5_13e2b3a769b /home/vagrant/shared/data/testlogs/files/5.log
3 : c1d635c2a270008 /var/log/aerospike/aerospike1.log
.
.
.
Log-analyzer> select 1 2 3
Log-analyzer>
Log-analyzer> list
Added Logs:
1 : MD5_0b32a78b084 /home/vagrant/shared/data/testlogs/files/6.log
2 : MD5_13e2b3a769b /home/vagrant/shared/data/testlogs/files/5.log
3 : c1d635c2a270008 /var/log/aerospike/aerospike1.log
Selected Logs:
MD5_0b32a78b084 /home/vagrant/shared/data/testlogs/files/6.log
MD5_13e2b3a769b /home/vagrant/shared/data/testlogs/files/5.log
c81635c2a270008 /var/log/aerospike/aerospike2.log
.
.
.