Reducing Search Friction with A.I.

“An important application of AI and machine learning in online marketplaces is the way in which potential buyers engage with the site and proceed to search for products or services,” Milgrom and Tadelis note.27 At Google, Facebook, and Amazon AI-powered search engines are trained to maximize what the provider believes to be the right objective.27 “Often this boils down to conversion, under the belief that the sooner a consumer converts a search to a purchase, the happier the consumer is both in the short and the long run,” say Milgrom and Tadelis.27 The rationale: “search itself is a friction, and hence, maximizing the successful conversion of search activity to a purchase reduces this friction.”27 

Although this is consistent with economic theory, which posits “search as an inevitable costly process that separates consumers from the products they want”27 this isn’t really the case. “Unlike the simplistic models of search employed in economic theory, where consumers know what they are looking for and the activity of search is just a costly friction, in reality, people’s search behavior is rich and varied,” claim Milgrom and Tadelis.27

In their paper Returns to Consumer Search: Evidence from eBay[i], Blake, Nosko, and Tadelis use “comprehensive data from eBay to shed light on the search process with minimal modeling assumptions.” Blake et al.’s data showed that consumers search significantly more than in previous studies, which were conducted with limited access to search behavior over time.125

“Furthermore, search often proceeds from the vague to the specific. For example, early in a search a user may use the query ‘watch’, then refine it to ‘men’s watch’ and later add further qualifying words such as color, shape, strap type, and more,” explain Blake et al.125 This behavior suggests that consumers aren’t looking specifically at first and are exploring their own tastes, and what product characteristics might exist, as part of their search process.125 Blake et al. showed that the average number of terms in a user’s query “rises over time, and the propensity to use the default ranking algorithm declines over time as users move to more focused searches like price sorting.”125

“These observations suggest that marketplaces and retailers alike could design their online search algorithms to understand search intent so as to better serve their consumers,” recommend Milgrom and Tadelis.27 Consumers in the exploratory phases of the search process, should be provided some general offerings to better learn their tastes as well as all available options in the market.27 Once the consumer shows the desire to purchase something in particular, the offering should be narrowed to a set of products that match the consumer’s preferences.27 “Hence, machine learning and AI can play an instrumental role in recognizing customer intent,” contend Milgrom and Tadelis.27 

Milgrom and Tadelis explain that, AI and machine learning not only helps “predict a customer’s intent, but given the large heterogeneity on consumer tastes, AI can help a marketplace or retailer better segment the many customers into groups that can be better served with tailored information.”27

Using AI for more refined customer segmentation, or even personalized experiences, does raise price discrimination concerns.27 “For example, in 2012 the Wall Street Journal reported[ii] that ‘Orbitz Worldwide Inc. has found that people who use… Mac computers spend as much as 30% more a night on hotels, so the online travel agency is starting to show them different, and sometimes costlier, travel options than Windows visitors see.”27 Whether these practices of utilizing consumer data and AI to adjust pricing helps or harms consumers is up for discussion, but economic theory states that price discrimination can either increase or reduce consumer welfare.27 “If on average Mac users prefer staying at fancier and more expensive hotels because owning a Mac is correlated with higher income and tastes for luxury, then Orbitz practice is beneficial because it shows people what they want to see and reduces search frictions. However, if this is just a way to extract more surplus from consumers who are less price sensitive, but do not necessarily care for the snazzier hotel rooms, then it harms these consumers,” contend Milgrom and Tadelis.27 Either way, price elasticity systems can be set up if brands to choose to set them up.



[i] Blake, Thomas, Nosko, Chris, and Tadelis, Steven. (2016), Returns to consumer search: Evidence from ebay. In Proceedings of the 2016 ACM Conference on Economics and Computation, pages 531–545. http://www.hbs.edu/faculty/conferences/2016-dids/Documents/Tadelis_BNT_search_EC_vST.pdf (Accessed 22 January 2019).

[ii] Mattolli, Dana. (2012). Wall Street Journal. On Orbitz, Mac users steered to higher pricier hotels. August 23, 2012. https://www.wsj.com/articles/SB10001424052702304458604577488822667325882 (Accessed 3 February 2019).

 

Creating a market for feedback

Besides the over-inflation of customer feedback as described above, another problem with customer feedback forums is the fact that few buyers even bother leaving feedback.27 “In fact,” Milgrom and Tadelis argue, “through the lens of mainstream economic theory, it is surprising that a significant fraction of online consumers leave feedback. After all, it is a selfless act that requires time, and it creates a classic free-rider problem.”27 Additionally, “because potential buyers are attracted to buy from sellers, or products, that already have an established good track record, this creates a ‘cold start’ problem,”27 i.e., new sellers with no feedback face a high barrier-to-entry because buyers are hesitant to try them out.27

Li et al. address this problem in their paper Buying Reputation as a Signal of Quality: Evidence from an Online Marketplace[i] by Using a unique and novel implementation of a market for feedback on the huge Chinese marketplace Taobao where they let sellers pay buyers to leave them feedback.”27 Of course, it might be concerning to allow “sellers to pay for feedback as it seems like a practice in which they will only pay for good feedback and suppress any bad feedback, which would not add any value in promoting trust.”27 However, Milgrom and Tadelis explain that “Taobao implemented a clever use of NLP to solve this problem: it is the platform, using an NLP AI model, that decides whether feedback is relevant and not the seller who pays for the feedback.”27 “Hence, the reward to the buyer for leaving feedback was actually managed by the marketplace, and was handed out for informative feedback rather than for positive feedback,” note Milgrom and Tadelis.27

“Specifically, in March 2012, Taobao launched a ‘Rebate-for-Feedback’ (RFF) feature through which sellers can set a rebate value for any item they sell (cash-back or store coupon) as a reward for a buyer's feedback,” says Milgrom and Tadelis.27 Sellers who choose this option guarantee that the rebate will be transferred from the seller's account to a buyer who leaves high-quality feedback that is, most importantly, informative about the purchased product, rather than whether the feedback is positive or negative.27 “Taobao measures the quality of feedback with an NLP algorithm that examines the comment's content and length and finds out whether key features of the item are mentioned,” explains Milgrom and Tadelis.27 The marketplace actually manages “the market for feedback by forcing the seller to deposit at Taobao a certain amount for a chosen period, so that funds are guaranteed for buyers who meet the rebate criterion, which itself is determined by Taobao.”27

Taobao wanted to promote more informative feedback, but as Li et al. note, “economic theory offers some insights into how the RFF feature can act as a potent signaling mechanism that will further separate higher from lower quality sellers and products.”123

Building upon the work of Philip Nelson in his influential article Information and Consumer Behavior[ii] that suggested advertising acts as a signal of quality, say Milgrom and Tadelis suggest that, “According to the theory, advertising — which is a form of burning money — acts as a signal that attracts buyers who correctly believe that only high-quality sellers will choose to advertise.”27 “Incentive compatibility is achieved through repeat purchases: buyers who purchase and experience the products of advertisers will return in the future only if the goods sold are of high enough quality,” argue Milgrom and Tadelis.27 “The cost of advertising can be high enough to deter low quality sellers from being willing to spend the money and sell only once, because those sellers will not attract repeat customers, and still low enough to leave profits for higher quality sellers. Hence, ads act as signals that separate high quality sellers, and in turn attract buyers to their products,” state Milgrom and Tadelis.27

Li et al. argue that Taobao’s “RFF mechanism plays a similar signaling role as ads do, which can be seen as signals that separate high quality sellers, and in turn attract buyers to their products.”123 Assuming “consumers express their experiences truthfully in written feedback, any consumer who buys a product and is given incentives to leave feedback, will leave positive feedback only if the buying experience was satisfactory.”27 Li et al. believe that a seller will offer RFF incentives to buyers if he or she expects positive feedback, which usually only happens if the seller provides a high quality item and/or service.27 “If a seller knows that their goods and services are unsatisfactory, then paying for feedback will generate negative feedback that will harm the low-quality seller,” claim Milgrom and Taledis.27 “Equilibrium behavior,” Milgrom and Tadelis contend, “implies that RFF, as a signal of high quality, will attract more buyers and result in more sales.”27 “The role of AI was precisely to reward buyers for information, not for positive feedback,” state Milgrom and Tadelis27, and that is as it should be. 

Li et al. analyzed data “from the period where the RFF mechanism was featured, and confirmed that first, as expected, more feedback was left in response to the incentives provided by the RFF feature.”123 Li et al. also discovered that “the additional feedback did not exhibit any biases, suggesting that the NLP algorithms used were able to create the kind of screening needed to select informative feedback.”123 Li et al. conclude that, “the predictions of the simple signaling story were borne out in the data, suggesting that using NLP to support a novel market for feedback did indeed solve both the free-rider problem and the cold-start problem that can hamper the growth of online marketplaces.”123



[i] Li, L.I., Tadelis, S., and Zhou, X. (2016). Buying Reputation as a Signal of Quality: Evidence from an Online Marketplace. NBER Working Paper No. 22584.

[ii] Nelson, P. (1970). Information and Consumer Behavior. Journal of Political Economy, 78(2), 311-329. Retrieved from http://www.jstor.org/stable/1830691

NLP

In their article How Artificial Intelligence and Machine Learning Can Impact Market Design, Paul R. Milgrom and Steve Tadelis give some interesting use cases for NLP. Such online marketplaces as eBay, Taobao, Airbnb, along with many others have seen exponential growth since their inception because they provide “businesses and individuals with previously unavailable opportunities to purchase or profit from online trading.” Besides the new marketplaces created for these wholesalers and retailers, “the so called ‘gig economy’ is comprised of marketplaces that allow individuals to share their time or assets across different productive activities and earn extra income.”

“The amazing success of online marketplaces was not fully anticipated,” Milgrom and Tadelis surmise, “primarily because of the hazards of anonymous trade and asymmetric information. Namely, how can strangers who have never transacted with one another, and who may be thousands of miles apart, be willing to trust each other?” “Trust on both sides of the market is essential for parties to be willing to transact and for a marketplace to succeed,” claim Milgrom and Tadelis. eBay’s early success is often attributed to its innovative feedback and reputation mechanism, which has been replicated by practically every other marketplace that came after eBay.Milgrom et al. believe that these online feedback and reputation mechanisms provide a modern-day version of more ancient reputation mechanisms used in the physical marketplaces that were the medieval trade fairs of Europe.[i]

The problem for Milgrom and Tadelis is that “recent studies have shown that online reputation measures of marketplace sellers, which are based on buyer-generated feedback, don’t accurately reflect their actual performance. A growing body of research reveals that “user-generated feedback mechanisms are often biased, suffer from ‘grade inflation,’ and can be prone to manipulation by sellers.” “For example, the average percent positive for sellers on eBay is about 99.4%, with a median of 100%. This causes a challenge to interpret the true levels of satisfaction on online marketplaces,” state Milgrom and Tadelis.

For Milgrom and Tadelis, a natural question emerges: “can online marketplaces use the treasure trove of data it collects to measure the quality of a transaction and predict which sellers will provide a better service to their buyers?”27 After all, these online marketplaces and gig-economy sites collect vast amounts of data as part of the process of trade. The millions of transactions, searches and browsing that occur in these marketplaces every day could be leveraged to create an environment that promotes trust, similar to the way institutions emerged in the medieval trade fairs of Europe that helped foster trust. Milgrom and Tadelis believe that AI can be applied to these marketplaces to help create a more trustworthy and better buying experience to consumers.

“One of the ways that online marketplaces help participants build trust is by letting them communicate through online messaging platforms,” explain Milgrom and Tadelis. On eBay, buyers question sellers about their products, “which may be particularly useful for used or unique products for which buyers may want to get more refined information than is listed.” Airbnb also “allows potential renters to send messages to hosts and ask questions about the property that may not be answered in the original listing.”

Using NLP, “marketplaces can mine the data generated by these messages in order to better predict the kind of features that customers value.”27 However, Milgrom and Tadelis claim, “there may also be subtler ways to apply AI to manage the quality of marketplaces.”27 The messaging platforms are not only restricted to pre-transaction inquiries, they also provide both parties the ability to send messages to each other post-transaction.27 The obvious question that emerges for Milgrom and Tadelis is, “how could a marketplace analyze the messages sent between buyers and sellers post the transaction to infer something about the quality of the transaction that feedback doesn't seem to capture?”

This question was posed and answered in the paper Canary in the e-commerce coal mine: Detecting and predicting poor experiences using buyer-to-seller messages[ii] by Masterov et al. Milgrom and Tadelis explain27:

“By using internal data from eBay’s marketplace. The analysis they performed was divided into two stages. In the first stage, the goal was to see if NLP can identify transactions that went bad when there was an independent indication that the buyer was unhappy. To do this, they collected internal data from transactions in which messages were sent from the buyer to the seller after the transaction was completed and matched it with another internal data source that recorded actions by buyers indicating that the buyer had a poor experience with the transactions. Actions that indicate an unhappy buyer include a buyer claiming that the item was not received, or that the item was significantly not as described, or leaves negative or neutral feedback, to name a few.” 

The simple NLP approach Milgrom and Tadelis use “creates a ‘poor-experience’ indicator as the target (dependent variable) that the machine learning model will try to predict, and uses the messages’ content as the independent variables.”27  In its simplest form and as a proof of concept, a regular expression search was used that included a standard list of negative words such as ‘annoyed,’ ‘dissatisfied,’ ‘damaged,’ or ‘negative feedback’ to identify a message as negative,” explain Milgrom and Tadelis.27 Messages void of these designated terms were considered neutral.27  Using this classification, the researchers grouped transactions into three distinct types: “(1) No post-transaction messages from buyer to seller; (2) One or more negative messages; or (3) One or more neutral messages with no negative messages.”27

In the second stage of the analysis, using the fact that negative messages are associated with poor experiences, Masterov et al. constructed a novel measure of seller quality based on the idea that sellers who receive a higher frequency of negative messages are bad sellers.122 According to Masterov et al., the measure, which is “calculated for every seller at any point in time using aggregated negative messages from past sales, and the likelihood that a current transaction will result in a poor experience,”122 is a monotonically increasing relationship.122 

This simple exercise shows that, using a marketplace’s message data and a simple NLP procedure, businesses can predict which sellers will create poor experiences better than one inferred from highly inaccurate and wildly inflated feedback data.27

Of course, eBay is not unique in allowing “parties to exchange messages and the lessons from this research are easily generalizable to other marketplaces.”27 “The key is that there is information in communication between market participants, and past communication can help identify and predict the sellers or products that will cause buyers poor experiences and negatively impact the overall trust in the marketplace,” conclude Milgrom and Tadelis.27



[i] Milgrom, P.R., North, D.C. and Weingast, B.R. (1990). The role of institutions in the revival of trade: The law merchant, private judges, and the Champagne fairs,” Economics and Politics, 2(1):1-23.

[ii] Masterov, D. V., Mayer, U. F., and Tadelis, S. (2015) “Canary in the e-commerce coal mine: Detecting and predicting poor experiences using buyer-to-seller messages,” In Proceedings of the Sixteenth ACM Conference on Economics and Computation, EC '15, pp81-93.