The criteria for making a choice here are all so subjective.
A 20" wheel and tire have more "unsprung weight," standing alone a negative in a serious track car or race car.
On the other hand, a 20" tire has a stiffer sidewall, a positive attribute in a serious track or race car (simply put, less sidewall deflection and thus higher g-force potential).
A 19" tire may "communicate" slightly better in track-speed cornering as the wheel begins to slide under load. This is more or less important depending on your skill level. The price of that communication is diminished mechanical grip.
These are almost irrelevant factors in a moderately driven road car. In that context, you are mainly weighing looks against wheel damage issues.
I have a racing background and retain a strong high performance bias even for street driving. I am relatively careful around potholes and bumps. I like the look of my 20's and the feel on the street and track. I have had no issues with wheel damage. I would personally not change to a smaller size, especially 18s, where availability of tires is an issue. Also, 18s change the gear ratios ( not so with 19s, which have the same overall diameter as 20s). Again, this is so subjective and debatable.