Nick Gibb spoke out ahead of the government publishing new guidance for schools, expected to address internet safety, social media and online gaming.
It is expected to say children should be taught to limit the amount of time they spend online.
Schools have the power to ban phones from being taken on to the premises.
But government policy is that it is the responsibility of head teachers to determine whether this is appropriate.
Patsy Kane, executive head teacher at the Education and Leadership Trust in Manchester, said Mr Gibb’s plan was “missing the point on just how fantastically useful mobile phones can be for learning.”
She told BBC Breakfast her multi-academy trust encourages the responsible use of smartphones during lessons and that teachers do remove the devices if they are used inappropriately during school time.
She added: “There’s a fantastic range of apps now for revision – and the students are really motivated to use them.”
The trade union that represents head teachers has expressed its scepticism about outright bans.
But Mr Gibb said: “Many schools have already taken the decision to ban mobile phones from the classroom.
“While this is clearly a matter for the head teacher, my own view is that schools should ban their pupils from bringing smartphones into school or the classroom.”
Katie Ivens, of the Campaign for Real Education, said she supported a classroom ban but that pupils should be allowed to carry phones to and from school for safety.
What is known as “relationships education” is to become compulsory from September 2020, with schools being encouraged to teach it from September of this year.
Last summer, the government published draft guidance about how this could be implemented.
“Children and young people are growing up in an increasingly complex world and living their lives seamlessly on and offline,” it said.
But it acknowledged that while this presented “many positive and exciting opportunities” there were also “challenges and risks”.
Since the publication of the document, which said “pupils should be taught about the benefits of balancing time spent on and offline”, there have been more than 11,000 responses to it and new guidance is expected soon.
“Children should not be spending hours and hours on their smartphones or iPads. There are obviously huge benefits to the internet and there’s nothing intrinsically damaging about spending time online,” Mr Gibb said.
“But if the time children spend using social media or playing computer games becomes excessive, it drives out time for them to talk to their parents, exercise, do their homework or play with friends.
“It eats into the amount of sleep and rest children have, resulting in their coming into school the next day tired and unable to concentrate.”
The minister, who has spent more than six years at the Department for Education, added: “Ensuring children can regulate their own use of smartphones and social media is becoming an increasingly important life skill for them to learn.”
But the NAHT union, which represents school leaders, has said banning pupils from taking a mobile phone to school can be counterproductive.
Giving evidence to the Commons Science and Technology Committee in October last year, the union’s senior policy adviser Sarah Hannafin said: “Mobile phone bans certainly work for some schools but there isn’t one policy that will work for all schools.
“Outright banning mobile phones can cause more problems than it solves, driving phone use ‘underground’ and making problems less visible and obvious for schools to tackle.”
She added: “Ultimately, schools work to prepare young people for the outside world, giving them the awareness and strategies to responsibly monitor their own screen use and the ability to identify and deal with any negative impacts or problematic content they encounter.”