Insurance premium principle includes the expected loss plus a risk loading factor to cover the loss from adverse claim experience and generate a profit. This paper considers a stochastic differential game with multiple insurers who are competing to sell insurance contracts by controlling their insurance premium. The existing works on this competitive premium problem mainly follow the development of retail pricing models, but fail to consider the randomness of payoffs by selling insurance contracts. The present paper aims to fill in this gap with the large body of literature on insurance surplus process modeling. Specifically, we model the surplus per policy unit by the diffusion approximation to the classical Cramér-Lundberg model. The risk exposure of an insurer (i.e., the number of policies) is assumed to be impacted (linearly) by all insurers in the market. Closed-form Nash equilibrium premium strategies are solved for insurers who are aiming to maximize their expected terminal exponential utilities. To investigate the robustness of equilibrium premium strategies, we further allow insurers to perceive different levels of ambiguity towards the underlying aggregate claim amount process. Closed-form expression for such robust premium strategies are obtained as well, and comparative statics analysis for the model parameters is implemented.