I practice and practice and practice with it. I don't use the sights. I "point and shoot." Humans are very good at pointing to things with some accuracy.
I fire rapidly, "sighting," straight down my dominant arm. Similar to your circumstance, I wear light clothing out and about in the yard. I carry the BG380 in a Kydex holster IWB on my running shorts and a thin T-shirt.
I can't pull the gun without the Kydex coming out with it, so I've practiced stripping the holster with my left hand as part of the draw. I carry with 7 rounds in the mag and one in the chamber. There's no safety.
While the BG380 is an ugly beast for target practice or long distance defense, its small size and light weight work well for coming and going out in the yard.