Silence is as important as the music but it can also be misused as you point out. It's finding the right balance between the two. Sometimes directors are surprised when the composers tell them to avoid music in a certain place, thinking they'll always want to add music to a scene. But a good film composer knows when to leave silence to do its magic too.