Query messages using full-text search

All messages sent to elmah.io, are indexed in Elasticsearch. Storing messages in a database like Elasticsearch opens up a world of possibilities. This article explains how to query your log messages using full-text search, Search Filters, and Lucene Query Syntax.

The easiest approach to start searching your log messages is by inputting search terms in the Search field on elmah.io:

Full-text query

We don't want to get into too much detail on how full-text works in Elasticsearch. In short, Elasticsearch breaks the query into the terms implement and icontroller and tries to match all log messages including those terms. Full-text search work on analyzed fields in Elasticsearch, which means that wildcards and other constructs are fully supported.

Full-text queries work great. when you want to do a quick search for some keywords like part of an exception message or stack trace. Remember that the entire log message is search, why a search for 500 would hit both log messages with status code 500 and the term 500 in the stack trace.

Search Filters

Search filters are built exclusively for elmah.io. They are built on top of Lucene Query Syntax (which we'll discuss in a minute), but much easier to write. Search filters are available through either the Add filter button below the search field or using various links and icons on the elmah.io UI.

Let's say we want to find all errors with a status code of 500:

Search filters

Adding the two filters is possible using a few clicks.

As mentioned previously, search filters are available throughout the UI too. In this example, a filter is used to find messages not matching a specified URL:

Search filter by URL

Search filters can be used in combination with full-text queries for greater flexibility.

Lucene Query Syntax

Elasticsearch is implemented on top of Lucene; a high-performance search engine, written entirely in Java. While Elasticsearch supports a lot of nice abstractions on top of Lucene, sometimes you just want close to the metal. This is when we need to introduce you to Lucene Query Syntax. The query syntax is a query language similar to the WHERE part of a SQL statement. Unlike SQL, the query syntax supports both filters (similar to SQL) and full-text queries.

All Lucene queries are made up of strings containing one or more terms and operators:

term AND term OR (term AND NOT term)

Instead of AND, OR, and NOT you can use operators known from C#:

term && term || (term && !term)

While AND, OR, and NOT speak for themselves, terms need a bit of explanation. A term can be a single term or a phrase. We've already seen two single terms in the full-text search example. The query in the example corresponds to this Lucene query:

implement AND IController

Looking at term phrases, things get interesting. With phrases, you can query on specific fields, perform range queries, and much more. Examples are worth a thousand words, why the rest of this document is examples of frequently used queries. If you think that examples are missing or have a problem with custom queries, let us know. We will extend this tutorial with the examples you need.

Find messages with type

type:"System.Web.HttpException"

Find messages with status codes

statusCode:[500 TO 599]

Find messages with URL and method

url:"/tester/" AND method:get

Find messages with URL starting with

url:\/.well-known*

The forward slash, in the beginning, needs to be escaped, since Lucene will understand it as the start of a regex otherwise.

Find messages by IP

remoteAddr:192.168.0.1

Find messages by IP's

remoteAddr:192.68.0.*

The examples above can be achieved using Search Filters as well. We recommend using Search Filters where possible and falling back to Lucene Query Syntax when something isn't supported through filters. An example is using OR which currently isn't possible using filters.

Field specification

As already illustrated through multiple examples in this document, all log messages consist of a range of fields. Here's a full overview of all fields, data types, etc. The .raw column requires a bit of explanation if you are not familiar with Elasticsearch. Fields that are marked as having .raw are indexed for full-text queries. This means that the values are tokenized and optimized for full-text search and not querying by exact values. In this case, a special raw field is created for supporting SQL like WHERE queries. To illustrate, searching for a part of a user agent would look like this:

userAgent:chrome

And searching for a specific user agent would look like this:

userAgent.raw:"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36"

Here is the full set of fields:

Name Type .raw Description
applicationName string Used to identify which application logged this message. You can use this if you have multiple applications and services logging into the same log.
assignedTo string The id of the user assigned to the log message. elmah.io user IDs are not something that is published on the UI anywhere why this field is intended for the My errors dashboard only.
browser string A short string classifying each log message to a browser. The value can be one of the following: chrome, safari, edge, firefox, opera, ie, other.
correlationId string CorrelationId can be used to group similar log messages into a single discoverable batch. A correlation ID could be a session ID from ASP.NET Core, a unique string spanning multiple microservices handling the same request, or similar.
country string An ISO 3166 two-letter country case in case we could resolve a country from the log message.
detail string A longer description of the message. For errors, this could be a stack trace, but it's really up to you what to log in there.
domain string The domain name if it could be resolved from the server variables.
hash string A unique hash for a log message. The hash is used for multiple things on elmah.io like the new detection.
hidden boolean A boolean indicating if a log message has been hidden through the UI, the API, or a hide rule.
hostName string The hostname of the server logging the message.
isBot boolean A boolean indicating is a log message is generated by an automated bot or crawler. This flag is set manually through the UI.
isBotSuggestion boolean A boolean indicating if the log message looks to be generated by an automated bot or crawler. Unlike the isBot field, this field is automatically set using machine learning and is available on the Enterprise plan only.
isBurst boolean A boolean indicating if the log message is a burst. Log messages are automatically marked as burst if we have seen it more than 50 times during the retention period of the purchased plan.
isFixed boolean A boolean indicating if the log message has been marked as fixed. Log messages can be marked as fixed from the UI.
isHeartbeat boolean A boolean indicating if the log message is logged from the elmah.io Heartbeats feature.
isNew boolean A boolean indicating if we have seen this unique log message before.
isSpike boolean A boolean indicating if the log message is logged from the elmah.io Spike feature.
isUptime boolean A boolean indicating if the log message is logged from the elmah.io Uptime Monitoring feature.
message string The textual title or headline of the message to log.
messageTemplate string The title template of the message to log. This property can be used from logging frameworks that support structured logging like: "{user} says {quote}". In the example, titleTemplate will be this string and the title will be "Gilfoyle says It's not magic. It's talent and sweat".
method string If the log message relates to an HTTP request, you may send the HTTP method of that request. If you don't provide us with a method, we will try to find a key named REQUEST_METHOD in serverVariables.
os string A short string classifying each log message to an operating system. The value can be one of the following: ios, windows, android, macos, linux, other.
remoteAddr string The IP address of the user generating this log message if it can be resolved from server variables.
severity string An enum value representing the severity of this message. The following values are allowed: Verbose, Debug, Information, Warning, Error, Fatal.
source string The source of the code logging the message. This could be the assembly name.
statusCode number If the message logged relates to an HTTP status code, you can put the code in this property. This would probably only be relevant for errors but could be used for logging successful status codes as well.
time date The date and time in UTC of the message. If you don't provide us with a value, this will be set the current date and time in UTC.
type string The type of message. If logging an error, the type of the exception would go into type but you can put anything in there, that makes sense for your domain.
url string If the log message relates to an HTTP request, you may send the URL of that request. If you don't provide us with an URL, we will try to find a key named URL in serverVariables.
user string An identification of the user triggering this message. You can put the user's email address or your user key into this property.
userAgent string The user agent of the user causing the log message if it can be resolved from server variables.
version string Versions can be used to distinguish messages from different versions of your software. The value of the version can be a SemVer compliant string or any other syntax that you are using as your version numbering scheme.

This article was brought to you by the elmah.io team. elmah.io is the best error management system for .NET web applications. We monitor your website, alert you when errors start happening, and help you fix errors fast.

See how we can help you monitor your website for crashes Monitor your website