Elizabeth Banks, Amy Adams and Sarah Paulson among stars fanning out at Taylor Swift's shows