Solskjaer has come in for criticism in recent weeks after being outclassed in Premier League home losses to Liverpool and Manchester City.
The Old Trafford club have dropped to sixth on the table and are already nine points behind leaders Chelsea.
Manchester United have continued to back Solskjaer in his difficult spells as manager but Ferdinand feels his former colleague is not up to the task and should move on.
"We started this season with such hope but also the transfer window that we had, the excitement, we were all sitting there going 'this is our time now'," Ferdinand – who won six Premier League trophies and the Champions League during his time at United – said on his Five with Vibe podcast.
"Ole was brought in to get us to that point. He was brought in to get the squad in-tact, get the players in to make us challenge for the league.
"I sit here now and I don't think we can challenge for this league. I look at our team every week wondering what are we gonna do tactically?
"I don't see any philosophy or identity in the Man United way of playing whatever that should be from the management. I sit here looking confused at the team.
"I was always deep down a bit sceptical, could he take us on to be champions? I wasn't fully convinced – I wasn't sure he could do that.
"But the showing with the squad that he accumulated, to the beginning of this season – and what I've seen this season – I just feel it might be about the time now for the baton to be handed over to somebody else who can take us on now.
"And I think Ole would leave now with his head held high because, because from when he came in to where he got us at the beginning of the season I think it's been positive."
