This is a question I was asked about a week or so ago. I was able to dig deep enough into my memory bank to come up with an answer, backed up with a quick Google search. Yellow Poplar weevil is the culprit. This weevil not only likes to chew on poplar leaves, but also sassafras and magnolia leaves. Both the larval and adult stages of the weevil enjoy a good meal of poplar leaf.
Normally the yellow poplar weevil is not considered a serious pest, but in the 1960’s the US Forest Service reported damage to yellow poplar trees in large areas of the Appalachians. There is typically no reason for concern until trees are defoliated several years in a row.