Beat the egg, milk, and the melted butter together in a mixing bowl.
Add the chopped peaches and vanilla.
In a separate large mixing bowl, combine the flour, baking powder, salt, sugar, cinnamon, and nutmeg.
Make a well in the center of the flour mixture and add the milk and peaches mixture. Stir together just until blended. DO NOT OVERMIX.
Preheat oil in a skillet to 350 degrees.
Use a tablespoon or an ice cream scoop for larger fritters and drop mixture into the hot oil.
Fry to golden brown on all sides.
Turn the fritters over to brown evenly.
Dust with powdered sugar while hot.
Serve with peach butter.