In this task you need to evaluate the relevance of webpages in the context of commercial queries.

About the task

Commercial query is a query directly related to a vendor or a product submitted by user.

Commercial queries can contain a name of:

• a product category ([mixers], [vacuum cleaner], [clothes for toddlers], [goods for sport] etc.);
• a product manufacturer ([Nike], [Nescafe], [xiaomi] etc.);
• product category and manufacturer ([nespresso coffee machine], [Adidas boots], [pampers pants] etc.);
• a product model ([asus zenbook 14 ux433], [Candy CS C9LG-07] etc.)
• a store ([laptops walmart], [find flowers amazon.com], [6pm choose dresses] etc.);

Note the queries can contain supplementary words (markers) like "shop/shop online", "buy", "price" etc.

So mostly you need to evaluate:

• shopping pages (webpages that allow users to make purchases online, such as online stores). In this task we take into account shops in the USA or global shops only.
• official pages of companies, where they present their goods;
• pages with information about different goods.

Relevance scale and principles of evaluation

Relevance scale consists of 2 marks: Relevant and Irrelevant. To make the right choice, the main thing is you should take into account the content of the webpage. You should evaluate the page in general.

Relevant document corresponds to a query and provides the requested information. The document is Relevant Plus when it is completely dedicated to the subject of the query, if it gives a complete answer to the user’s query or it satisfies his needs in the best possible way. For example, you should get a set of objects for queries, which contain a name of a product category/product manufacturer etc. to use this mark. 

If the query contains something specific (a product model or specific name), you need to ensure the page is totally dedicated to this good. Note that the page should also correspond both to a query’s language and to a query's region. 

The page can also be indirectly related to the query or highly narrowing or expanding its meaning (minor objects). For example, the query is [samsung galaxy s21], and the page contains all the models of Samsung smartphones, or the query is [laptops], and you see just one object instead of a set of objects - this is also relevant. 

Irrelevant page doesn't contain the object of the query at all or its part is insufficient in comparison with the main contents of the page (so-called "random entry"). For example, the query is [black skirt], and you see information about black dresses only.