Schwartzman was born in Los Angeles, California, the son of actress Talia Shire (née Coppola) and the late producer Jack Schwartzman. Many other members of Schwartzman's family are involved in film: he is the nephew of Francis Ford Coppola, cousin of Nicolas Cage, Sofia Coppola, Roman Coppola and Christopher Coppola, and grandson of Italia Coppola (née Pennino) and Carmine Coppola (the source of his old stage name). His brother is Jason Schwartzman, lead singer of the band Coconut Records (also an actor and former drummer of the band Phantom Planet) and his half-brother is cinematographer John Schwartzman. Schwartzman is of Jewish descent on his father's side and of Italian descent on his mother's side.