A back up has to be concealable, which limits the caliber. The smallest 9mm is not suited for ankle or thich holder or pocket holster, unless you are wearing cargo pants. It depends on the climate and time of year. In south Georgia it's already hitting 80+ in the heat of the day. Casual dress that I wear consists of Dockers and a nice Polo type shirt. I can't even carry "my carry" in those clothes.
Working in a law office I have a handgun in a quick entry safe bolted to the inside wall of my desK. Therein is a .40 cal. On my belt is a "boxter holster" looking like a Blackberry carrier. Inside is my Ruger LCP .380. Remember the .380 is aka 9mm kurz, the mainstay of European police and secret service agents. It is a 9mm in a short case and with proper charge and bullet size is an adequate piece as long as you hit your target. Commonly known as the James Bond pistol. He carried a Walther .380/9mm kurz.
In my vehicle is a .45 acp, my weapon of choice. Unfortunately I have trouble concealing it when I visit another law office or police facility or do a scene investigation. Therefore I always have my Blackberry in .380.
In a nutshell; you do the best you can with the situation you are in. I consider the .380 as the smallest self defense weapon. But my holster making friend, Guru, says: better to have a .22 mag in your pocket than a .357 back in your vehicle. Think about it.
Whatever you carry: practice, practice, practice. Make damn sure you can hit your target. a .50 BMG isn't worth a **** if you miss.
If you are going to carry a backup using one that takes the same ammo/mags as your primary gun makes sense to me but like Whompuss said sometimes thats not possible. My carry guns are a Glock 26 and a NAA 22mag sometimes I only carry the 22. If you are going to carry everyday you have to find a gun/holster combo that works for the way you dress and doesn't get in the way of your daily activities. Sometimes thats hard to do with one gun.
I have always wondered about the need for a back-up gun for someone who is not an 'on-duty' LEO. The back-up weapon I carried was a Cold Steel 5" folder. Still do to this day because I would question how quickly you could get to another gun if your primary one malfunctioned. I think the fact that a civilian is generally only carrying for self defense purposes makes the need for 2 concealed firearms unlikely.
I would offer these alternatives thoughts though:
1. A small 'Derringer' type gun would conceal easily in a shirt pocket if you are already packing a regular carry on your waist...I have seen the .22 Mag ina 5 shot...you could get to it quickly and don't need another holster, but is it enough to defeat a threat in the situation like dethlokk describes?
What about one of these in .38? Is it a viable alternative?
But I also practise unjamming my carry gun (Glock27).
The drill is to turn the gun sideways with the ejector down and rack the slide a few times. If I were in the process of shooting and my gun jammed, I would rather quickly unjam it than try to pull out a different gun.
If I was an LEO and going in harms way I would find a way to pack a large caliber back up gun in the same caliber of my primary firearm.
As a citizen that does not hang out in bad neighborhoods and rarely experience any trouble, I will chose the lightweight path for my carry gun. No backup required other than an extra mag, a knife and my running shoes. Kel-tec 380 P3AT, most of the time. Sometimes a compact 45 acp.