This summer, the serve clock was introduced during the run-up to the 2018 US Open. It was designed in part to shorten the match with players such as Rafael Nadal and Novak Djokovic in mind — they both have lengthy serve rituals. Not serving before the clock strikes zero can lead to a code violation, then the call of a fault with the next violation. During the men’s final between Djokovic and Juan Martin del Potro on Sunday, the serve clock was on. Several times, Djokovic was still bouncing the ball when the clock ran down, but he didn’t receive a violation until the middle of the third set.

That call lies with the umpire, and she chose not to call it for 2½ sets. Several of her colleagues made the call early and often through the summer. Some umpires considered the moment before making the call on a serve clock violation, or any call that isn’t obvious. Umpires operate differently; it’s a normal part of tennis and not usually a big deal.

At the US Open women’s final on Saturday, a particular sort of officiating in tennis collided headfirst into an immoveable object — the long, fraught history Serena Williams has with this tournament. But before the debacle unfolded, which ended with Williams losing in straight sets after code violations that resulted in point and game penalties, and thunderous accusations of racism and sexism against umpire Carlos Ramos, there was some subjective officiating. Williams’s loss to Naomi Osaka had much less to do with racial or gender bias than with the inconsistent ways the sport is refereed.

Racket abuse is an automatic code violation, especially when it is as obvious as Williams’s was during the final against Osaka. Sometimes, though, if a player strikes the ground with their equipment, and it isn’t obvious that the racket has been destroyed and the player plays one more point with it before swapping it out, well, that one has been known to slide.

Coaching is also prohibited during grand-slam tennis. (It is permitted in women’s tennis during the regular season. It’s worth noting that Williams, and her sister, Venus, never choose to see their coaches during matches when it is allowed.) At majors, as during the rest of the year, this rule is commonly broken, by both men and women. Anyone who cared to call Nadal for it could do it easily. The same is true of Simona Halep and Djokovic. But it isn’t tightly observed, even when a coach is clearly gesturing at a player, as Williams’s coach was.

That is the trouble here. The observance of the rules of tennis depends on the person sitting in the chair. That person is not a robot. She may assert herself early in a match. Or he may be content to sit back, call only double-bounces and quiet the crowd when necessary, especially during a very important match. He may penalise a female player for changing her shirt on court or decide to get down from a chair and give a pep talk to the trailing player. (Those things happened in the past two weeks at this tournament.)

But ultimately, the strict adherence to rules comes down to an individual. It is like boxing or figure skating in this sense. One judge sees a point against the athlete; the other doesn’t. The difference between gold and silver lies in this grey space. The difference between a code violation and a whistle in the pocket lay on Saturday with Ramos. He has been known to mix it up with players over the years for violations they don’t like — from Djokovic to Nadal to Venus Williams.

Andy Roddick accused Ramos of “stupid umpiring” at the 2016 Olympic Games and suggested that he wanted to be “the star of the show”. Djokovic noted that during the Wimbledon quarter-final this year, he and his opponent both threw their racquets, but only Djokovic received a violation. Ramos also is one of the game’s highly regarded umpires, having called matches at all the grand slams in his 27-year career. What to make of that?

Serena Williams has a long and complicated history at the US Open. Her first slam happened in Queens. In 2004, the court umpire so badly botched her quarter-final against Jennifer Capriati that it led to the challenge system, where players can ask for a second look after a call has been made. In 2011, in the final, Williams let out one of her primal screams when she thought she had hit a winning shot against her opponent, Samantha Stosur, but technically Stosur was still running after the ball and could have retrieved it. In tennis, that’s called a hindrance, and she lost the point. Williams didn’t take it well. Most famously, in 2009, deep in a semi-final against Kim Clijsters, a line judge called her for a foot fault. She didn’t take it well, either. And then there was Saturday.

Perhaps Williams decided she would not stand for another unfair line call, foot fault, or a ruling that’s flat-out wrong — not without having a say. Over the years at the Open, she has tangled with officials, including Mariana Alves (2004), Eva Asderaki (2011), Louise Engzell and Shino Tsurubuchi (2009); with Brian Earley, tournament referee; and Donna Kelso, the tournament supervisor, who emerges only after the situation has gone to pot.

This year, it was Ramos. The circumstances around this controversy are not as obvious. Yes, umpire abuse is against the rules, but Roger Federer was not penalised on court during the tournament in 2009 after lacing into an umpire during an upset loss — although he was later fined.

Yes, Williams’s coach, Patrick Mouratoglou, was certainly trying to tell her something, but it’s not clear that she could see that, and her history suggests that she would not have been looking for coaching. Her history also suggests that she isn’t crazy to assume there was something else at work here.

The spectre of race has followed Williams throughout her career. At the Indian Wells tournament in 2001, she was slated to play her sister in the semi-finals, but Venus Williams backed out just before the match started. The Williams family later said that when Serena Williams next took to court in the final, racist taunts came from the crowd. It prompted a years-long boycott of the tournament for the sisters. That’s a part of the history Serena Williams carries around with her — it’s impossible not to. It’s not so easy to draw a line between Ramos’s ruling on Saturday and racism. The thing we know for sure is that Ramos is known to apply the rules fearlessly, regardless of gender and race.

Tennis officiating needs consistency, and that doesn’t mean that players can expect one thing from Ramos and another from Asderaki. Some rules in tennis are easy to enforce. A ball is either in or out. Coaching signals between players and coaches are against the rules; they should be called all the time. Right now, that doesn’t happen.

Tennis took a step towards eliminating human error when it introduced the challenge system, and the sport needs more of that when it comes to making calls that aren’t so obvious. This requires something most tennis fans, myself included, hate to consider: changing some of the rules. How can umpires spot every time a coach makes a signal, or how can they determine the difference between instruction and encouragement? They can’t. Racket abuse? Certainly Williams violated this rule and should have been called for it. But why keep these rules? When you have rules that are not and cannot be equally enforced, you are going to have accusations of racism, sexism or other forms of bias influencing the umpires. You’re going to have a debate about rules rather than a celebration of Osaka’s brilliant play.

Officiating often depends on the judgment of a single person. Combine that with a player who comes to a tournament almost expecting that the lines will not get called in her favour, and you get the women’s final at the 2018 US Open. It’s not about sexism, or even about race. It is about colour, though — the colour grey.

• Nafari Vanaski is a former journalist who has written about — and played — tennis for more than 12 years