Search Syntax Help

Last updated 4 years ago

Revision history

About Search Syntax

The search engine allows users to locate uploads via tags that are associated with an upload and some other metadata such as uploader and user count. It also permits chaining together specified tags and metadata to search for specific logical combinations, to allow more precise filtering. This guide explains the syntax and features of individual terms and then shows how they are combined into more complex queries.

1. About Search Syntax
2. Search Terms
   1. Tag Search Behavior
   2. Searching Through Other Fields
   3. Numeric Range Queries
   4. Date/Time Range Queries
   5. Supported Fields
3. Special Characters and Suffixes
   1. Wildcards
   2. Escaping Special Characters
   3. Approximate (Fuzzy) String Matching
4. Search Grammar: Term Operators and Combinations
   1. Expressions
   2. Summary Table
   3. Negation
   4. Commas and AND Expressions
   5. OR Expressions
   6. Compound Expressions
      1. Operator Precedence
      2. Defining Subexpressions with Parentheses
      3. Automatic Parentheses Escaping
5. Boosting Terms

Search Terms

Specific searches require the inclusion of search terms, which individually define the criteria expected of each upload result to be returned by the search engine.

Tag Search Behavior

Searching a single term is obvious: merely type in the term you want. By default, the term you use will be searched among the indexed image tags and aliases. Thus, a search for canine would, as you may surmise, result in all appropriately tagged and indexed pictures of canine tagged images. Aliases are also indexed, so a search for the tag alias smile is the same as one for smiling.

The default tag search has particular aspects associated with it for your convenience. For tag searches, case is insensitive. This means capitalization is irrelevant for queries. For example, the search queries Canine and canine will share the same result set.

Searching Through Other Fields

Other fields are also indexed, and you can search them using the namespace convention that is also used by tags. Namely, one enters the field name followed by a colon, and finally, the target value. For example, to search for images with a width of 1920, we would search within the width field and so construct the query width:1920. If a tag with namespace were to share the namespace with a given field, it can still be queried via quoting or escaping.

Numeric Range Queries

Numeric fields in particular support queries for ranges of possible values. A qualifier can be added to the end of the field name with a single period to indicate desired results that are greater than or less than the supplied value; the value can be optionally included, too. To find images with a score greater than 100, we would enter score.gt:100. For an inclusive search of scores greater than or equal to 100, we would instead enter score.gte:100. The following table enumerates the supported qualifiers.

Date/Time Range Queries

Date and time values are specified using a tweaked subset of the ISO 8601 standard. A full date is specified by four-digit year, followed by two-digt month and date, with each value delimited by a hyphen, i.e., "YYYY-mm-DD". Like in ISO 8601, one can specify just the month or even just the year, as long as the less precise information is included in left-to-right order without dangling hyphens. This is semantically interpreted as the range of the entire period (not just the first day of the month, etc.). For example, 2020-05 represents the entire month of May 2020.

Given a full date, a specification for the time of day can be added. To do so, separate the time with a T or space, followed by the hours, minutes, and seconds, each specified ' with two digits and separated by a colon, i.e., "HH:MM:SS". The hours follow a 24-hour clock. As with date values, one may alternatively specify entire minutes and even hours by truncating the value without a dangling colons. The value 2020-05-20 16 represents the entire hour of 4 PM on 20 May 2020 (UTC). The entire first minute can be specified with 2020-05-20 16:00.

By default, time follows international UTC ("Zulu") time. (In terms of the ISO 8601 standard, a Z suffix is implied.) One may specify an offset for local time by affixing a plus or minus sign, followed by the offset hours as two digits, a colon, and the offset minutes (usually 00), e.g., -04:00 for US Eastern Daylight Time (EDT). Note that unlike ISO 8601, this can be attached to dates as well as times, to ensure date boundaries fit the locale of interest. For example, 2020-05:00 represents the year of 2020 with an offset of minus five hours (US Eastern Standard Time).

Date/time range queries also accept range qualifiers. The gt and lt qualifiers omit everything matching the implied time range of the specified value, whereas gte and lte include the entirety of said time range.

The following examples are valid search queries.

Supported Fields

The following table enumerates all of the supported fields, with examples.

It is worth noting the absence of certain “fields” such as artist and oc. These are tag namespaces, not metadata, but they are functionally the same. Thus, a search for oc:freckles performs as expected.

Special Characters and Suffixes

Wildcards

Wildcards allow for matching with terms that begin with, end with, or contain a given string of characters, like wildcards used in file management. Two wildcards are recognized: the asterisk (or star) and the question mark.

A question mark matches to a single character in its place, including none. For example, *ox matches to uploads with any of the tags box, fox, and simply ox.

An asterisk matches to any character in its place. For example, *glasses can match to either glasses or sunglasses.

Escaping Special Characters

The use of special characters that modify search terms or exist outside search terms mandates a facility for “escaping” those characters, so that they are not excluded from search terms themselves. To use special characters within a search term, both of the conventional string escaping mechanisms are used: the backslash and quoting. The following are special characters and sequences that may need to be escaped:

• (
• )
• *
• ?
• - (when placed in front of a term)
• ! (when placed in front of a term)
• ,
• &&
• ||
• OR (if all-capitalized)
• AND (if all-capitalized)
• NOT (if all-capitalized)
• "
• \
• ~ (with fuzzy matching syntax)
• ^ (with boosting)

A backslash is placed in front of a special character (and can also be placed in front of a sequence like the ones in the preceding list). This forces a given character to be counted as part of the preceding or following term. In front of any other character, it effectively has no effect. For example, \-_- forces a search for the emoticon -_-, despite it following the syntax for negation if without the backslash. Also consider the search term frog \(hoof\), although parentheses have intuitive rules that do not make escaping them necessary in most cases. The backslash is a special character and thus must also be escaped; a literal backslash is indicated with \\.

The alternative to escaping is to simply surround the search query in double quotes ("), e.g., "frog (hoof)". When searching with a specified field, quotes must surround the field and colon as well, e.g., "width:1920". Eveything in quotes is together treated as a verbatim search term, with one exception. Note that the double quote character itself bounds the search term, so if it appears inside, it must be escaped with a backslash. All other uses of backslash are treated literally.

Approximate (Fuzzy) String Matching

The search engine backend, Apache Lucene, also enables so-called “fuzzy” string matching. Fuzzy string matching can be used with any literal search term, including the default tags field. A fuzzy match is specified using a similarity metric either ranging from 0 to 1.0 or a whole number. The whole number specifies an optimal string alignment edit distance, which is the maximum number of edits done to a string to match a given target, with an edit defined as a deletion, insertion, replacement, or switching two adjacent characters. One may alternatively define a similarity factor ranging from 0 to 1.0, with a 1.0 the least “fuzzy”. The derived edit distance is the length of the term sans the field name prefix, multiplied by the difference of unity minus the similarity factor, all rounded down. To specify either, a term is followed with a tilde followed by the edit distance or similarity factor. Note in both cases that Lucene caps the maximum edit distance at 2, as an optimization. Therefore, very large edit distances or small similarities will not behave as expected.

For example, equina~0.8 searches for uploads with tags that approximately match equina, with a similairty of 0.8. This is an edit distance of ⌊(1 − 0.8)(10)⌋ = 2. Note that uploads tagged equine are included in the result set. The utility of this is obvious: if you are unsure of a character or tag's exact spelling, you can use this as an aid, like a more manual and controlled version of Google's (in)famous spelling correction features.

Fuzziness can also be applied to numeric queries to specify a range. In this case, the fuzziness parameter is the magnitude above and below the specified number that will be included in the result set. For example, width:800~200 specifies images with a width ranging from 600 (800 − 200) to 1000 (800 + 200), inclusive.

Fuzzy matching can be freely applied to any term inside an expression.

Search Grammar: Term Operators and Combinations

Expressions

Terms can be combined to define a search query corresponding to a specific result set. These combinations are formulized as expressions that are constructed from terms, operators, and even other expressions, which are then called subexpressions. Expressions recognized by the search frontend are the negation of a term or subexpression, the requirement of any search term or subexpression, or the requirement of both search terms or subexpressions.

At its core, a search expression is either binary or unary. A binary expression consists of a term or subexpression, an operator indicating the type of expression, and another term or subexpression. Binary expressions can be “chained” by adding the operator followed by another term. A unary expression consists of the operator followed by a single term or subexpression. Both expression types and how to use subexpressions will be covered in the following sections.

Summary Table

Negation

Negation of a term or expression specifies that the the original term or subexpression must not match. The corresponding negation operator is unary, that is, applied to either a single term or to a subexpression. It is specified with the all-capitalized word NOT, a dash of the non-multi-chromatic variety (-), or an exclamation point (!). For example, -equine or NOT equine matches pictures that are not tagged with equine. In set theory terms, this is taking the complement of the original result set, that is, all uploads outside it.

Commas and AND Expressions

An expression that queries for images that meet all specified terms is a conjunction or AND expresssion. As in the past, you can query images that meet a list of terms by hooking the terms together with commas. For example, equine,canine results in pictures that contain both the fluttersy and canine tags. In set theory terms, the result set is the intersection of uploads tagged equine and uploads tagged canine.

Commas can be padded with spaces however you like. Unlike the past, commas are now plain AND operators, so they are more versatile. As will be discussed, they can be used in subexpressions and alongside the OR operator.

AND operators can also be expressed using && (derived from typical programming notation) or the all-capitalized word AND, e.g., feral && canine or feral AND canine. These forms, unlike the comma, require padding space on either side.

OR Expressions

A disjunction or OR expression requests for uploads that meet any of the specified search terms. This is markedly different from the aforementioned AND expression, which, to reiterate, mandates that all terms match. OR operators are expressed either with || (also a programming notation) or the all-capitalized word OR, e.g., feral || canine or feral OR canine. In set theory terms, the result set is the union of uploads tagged feral and uploads tagged canine. All forms of the OR operator require padding on either side.

Compound Expressions

Complex combinations of terms, and therefore search criteria, are possible by combining expressions together. Doing so effectively is analogous to arithmetic. Consider multiplication and addition (which in so-called Boolean alegra are respectively analogous to AND and OR operations). We can express an algebraic expression with multiplication and addition several ways. For three terms, A, B, and C, consider the expression A × B + C. Multiplication is evaluated before addition, so this expression is equivalent to (A × B) + C, in which case the order of operations is explicit.

Operator Precedence

Likewise, precedence is applied to determine the order in which chained OR, AND, and NOT operations are evaluated. The order of operations in the search syntax is as follows:

1. negation (NOT)
2. conjunction (AND)
3. disjunction (OR)

Consider the query smiling || equine && canine. In this example, equine && canine is evaluated first, as an implicit subexpression. Then, that result is OR'd together with smiling. Thus, the query instructs the engine to return uploads either tagged with smiling or tagged with both equine and canine. Note how if the OR expression smiling || equine were evaluated first, the result set would differ.

Defining Subexpressions with Parentheses

Returning to an earlier example with arithmetic, we can trump the order of operations using explicit subexpressions. This requires the use of delimiters that act as boundaries, and most often parentheses are used for this purpose. Hence, A × (B + C) forces B + C to be evaluated, and then multiplied with A, which is contrary to the order otherwise followed. Likewise, (smiling || equine) && canine instructs the search engine to return results that have either smiling or equine and always match the tag canine.

As was mentioned earlier, the unary NOT operator can be applied to parenthesized subexpressions. The semantics of this is analogous to applying it to a single term: a negated subexpression specifies uploads that do not adhere to what the subexpression specifies. For example, the query -(blood, grimdark) returns all uploads that are not tagged with both blood and grimdark. Uploads tagged with either of the two would be returned as long as they do not have both. Thus safe images with blood and grimdark material not showing blood would be included, yet the intersection of those two sets of images would be excluded, that is, images that are grimdark and contain blood.

Explicit subexpressions with parentheses allow for complex queries as they can be arbitrarily nested inside other subexpressions, to fine-tune the result set even more.

Automatic Parentheses Escaping

Finally, a footnote about paretheses is warranted. Traditionally, if an expression parser encounters an open parenthesis without a closing parenthesis, or if parentheses are swapped, an error is raised. This is indeed the case with the search engine, as highlighted in the search parsing error page. However, to a limited extent, a term can contain parentheses within. Parentheses are accepted within search terms as long as they are closed and do not cover the entire expression. The first limit is a heuristic to address the typical use of parentheses, and the latter arises from the legal use of parentheses to single out a term. Thus, the search frog (hoof) searches for uploads tagged with frog (hoof) ; however, the emoticon query ))B-( raises an error, while (q) effectively searches for q, instead. For the latter two examples, simply surround with double quotes to clarify your meaning to the search engine.

Boosting Terms

The search engine also allows the boosting of specific terms when sorting by relevance, so that uploads including or not including the term occur earlier or later in the results. Boosting is done by modifying a term's relevance score with a positive or negative value. This value is affixed to a term with a preceding caret (^) and with a positive or negative decimal number. For example, canine^1 || dog returns uploads tagged either with canine or dog, but when sorting by relevance descending, uploads with canine are prioritized. A negative value meanwhile reduces the relevance score and deprioritizes the affected term when sorting by relevance, e.g., canine^-1 || dog. Sorting options are found below the search box on this page and must be set to sort by relevance for boosting to take proper effect. Thus, in both cases, pictures with both tags will still appear first.