I can see your point, but I’ve learned from @jongolson that this exact reason is why an affiliate marketer needs to have an email list.
Without an e-mail list an affiliate marketer doesn’t have a business, probably just a bit of luck.
When building your list, you should also work on your personal branding, so that when you introduce your list to a new affiliate product you found, they are inclined to believe you and trust you.
Also you should be careful what products you share, in order to build that trust.
This way an affiliate marketer, just like a dropshipping business can build a brand and a group of loyal followers.