On digital labor platforms rating system is applied to individual workers—with tremendous consequences for those being evaluated
Anyone who buys goods or services online has come to rely on the guidance of the stars—that is, the five-star rating system that aggregates customer evaluations of everything from products to restaurants to nursing homes.
On digital labor platforms like Uber, TaskRabbit, and Upwork, that rating system is applied to individual workers—with tremendous consequences for those being evaluated.
“It misses a lot of the variety and texture that exists in a labor interaction,” says Hatim Rahman, assistant professor of management and organizations at Kellogg. And workers tend to grasp the downsides of an overly simplistic rating system very quickly, Rahman adds: “They start to say, ‘Hey, this doesn’t make sense; there are so many situations in which the rating someone’s leaving does not accurately capture what’s happening in the context of my work.’”
Perhaps workers would shrug off the low ratings they perceive to be unfair if it weren’t for the financial repercussions that can flow directly from those ratings. Uber, TaskRabbit, and Upwork all integrate customer ratings into the algorithms that determine workers’ visibility to customers, eligibility for incentives, and continued employment.
What these platforms have created, in effect, is a system where the traditional role of the manager has been fully outsourced to customers and their star-based ratings. As Rahman and his colleague, Lindsey Cameron at Wharton, write in a new paper, “the rise of this algorithmically mediated customer control to monitor and evaluate individuals has effectively given many workers a new, digital ‘boss.’”
Rahman and Cameron investigated how individual workers across digital labor platforms set out to resist the power of these ratings, from vetting potential customers ahead of time to dropping work mid-project to avoid a bad rating.
The researchers divided the gig worker–customer interaction into three stages—before, during, and after the task—and found that each stage generated distinct forms of worker actions intended to recover some of their control over the process of being rated.
With each subsequent stage, workers’ ability to resist unfair ratings diminished, the researchers found. And efforts to remain vigilant over a prolonged service encounter and try to maintain high ratings contributed to feelings of fatigue among workers.
The researchers emphasize that these patterns likely spell problems for the platforms, too. Customers’ interests aren’t necessarily aligned with the interests of the platforms, and customers are unlikely to be held accountable for their rating decisions.
“When you outsource this type of control to people who aren’t vested in your platform, there are always going to be gaps,” Rahman says. “It opens up areas of mismatch and opportunities for gaming the system.”Also read: Drivers' lawsuit claims Uber, Lyft violate antitrust laws
Embedding in the Gig Economy
For their analysis of digital labor platforms, Rahman focused on a site that connects freelancers with projects, which he pseudonymously called “FindWork,” while Cameron studied a rideshare platform, which she calls “RideHail.” (The platforms were anonymized to protect workers’ identities.)
The researchers investigated the platforms’ dynamics by immersing themselves in RideHail and FindWork. Cameron spent the three years between 2016 and 2019 as a worker for and customer of RideHail; Rahman devoted the four years between 2015 and 2019 to the same roles on the FindWork platform.
As part of their investigation, they conducted interviews with workers and customers, and drew from archival sources. This included anonymized data from FindWork chronicling private communications from 2013 and 2014 between freelancers and customers during projects, and RideHail’s website materials, as well as articles, social-media posts, YouTube videos, how-to guides, blogs, and discussion boards about the company.
Across sources and platforms, the researchers saw clear patterns emerge. As workers moved through the different stages of a job, they responded with resistance measures specific to each stage. And their own power to push back against customers and their ratings decreased as the stages progressed.
During the first stage, workers had the most latitude to enact covert resistance tactics since customers have little information about workers at this point and cannot yet rate them. Their strategies at this point often involved attempts to vet customers—perhaps by calling them with a question to assess their attitudes and therefore their propensity to dole out a low rating.
The researchers learned that drivers who suspect that a prospective rider is apt to offer few stars will sometimes cancel a ride preemptively. “I never start the trip for a passenger with a bad attitude because that means bad ratings,” said one RideHail driver. On FindWork, some freelancers contact a customer before beginning a project to demand five stars as a precondition to working with them.
“In a grocery store, for example, you don’t really control which customer comes to you in the checkout line,” Rahman says. “But in our context, we see workers trying to assert more agency, because they know how it could unfold—they’ve taken steps to figure out how to avoid bad customers.”
During the second stage, while workers were in the midst of completing a task, their power to resist a customer’s demand or complaint diminished—since both workers and customers knew that, eventually, the customer would be rating the worker.
Tactics at this stage include, for example, offering a discount for a high rating. Alternatively, some freelancers on FindWork asked customers to spread a single project over multiple contracts, so they had the opportunity to receive multiple high ratings from a single customer they know and trust. Another strategy was ending work prematurely: a worker who suspected that a customer interaction was going south would simply cancel the job and, along with it, the possibility of a low rating—even though that also meant not getting paid.
“In traditional settings, if you have a bad customer, you can at least talk to a manager. But with platforms outsourcing the middle-manager role completely to the algorithm, we see in this middle stage a very heightened sensitivity from workers to every interaction,” Rahman says. “They work to make sure it goes as perfectly as possible, and they try to mitigate the risks of even one low rating.”
In the third and final stage, when workers have the least recourse to push back against low ratings, they are left to resort to what the researchers called the “Hail Mary” strategies of either filing a dispute with the platform or giving a customer a low rating when they suspect the customer has done the same for them. These tactics are unlikely to be successful in getting the low rating removed or changed, but workers still try them given the importance of ratings for their platform success.
Also read: Gig economy helps India's women workers gain financial independence
Collectively Seeking Control
Rahman believes that the limitations of labor platforms’ rating systems and the frustration and fatigue they produce in workers are already starting to manifest as problems for the platforms themselves.
“Across the world, including in the U.S., we’re seeing some contestation about how the platforms are treating workers,” Rahman says, adding that former Uber and Lyft drivers are starting worker-owned cooperatives in New York that offer similar services—while allowing workers to retain more ownership over how their workplaces operate. “We’re definitely seeing some pushback toward the model.”
But the implications of the study extend far beyond the usual gig-platform suspects, Rahman says. In the paper, he and Cameron explain how even in traditional service settings, employers are integrating more technology to solicit customer ratings across every stage of their experience. They note that hospitals are requesting real-time feedback from patients about the care they receive, the scheduling process, parking, and food, while airlines are seeking evaluations from customers about their experience buying tickets, checking bags, and boarding.
“One thing our paper highlights is the need to rethink whether this model of outsourcing control to a customer and primarily using a five-star rating system for all types of work contexts actually makes sense,” Rahman says.
[This article has been republished, with permission, from Kellogg Insight, the faculty research & ideas magazine of Kellogg School of Management at Northwestern University]