I suppose the hardest losses for Federer have all come at the hands of Nadal, the 2008 French and Wimbledon finals clearly stand out, the French that year was a disaster for Roger, previously he'd been closer but Nadal just looked a different class that day. Whilst a far more competitive match and indeed lauded as the greatest of all time by many, the Wimbledon final that year was a huge loss for Roger, from there he would quickly lose the world number 1 rank to Rafa later in the year and would be left to ponder whether Rafa was on his way to beating his varied records. Not long after the 2009 Australian final was much the same, as Roger realised he wasn't safe on even the hard courts going up against Rafa, particularly going into the final far fresher and having not lost in a GS final on hard courts, all indications pointed to reasonably easy win for Roger and instead Rafa once again bested him. Those are my considerations for his three hardest losses in GS tennis.
Join the group
Henman Wimbledon 2001
Djokovic USO 2010,2011,2015