With the emergence of machine learning within insurance, actuaries have many flexible tools at their disposal to improve the predictive performance of their pricing models. A major inconvenience with the main algorithms for machine learning regression (gradient boosting machines and neural networks) is ignoring the uncertainty associated with the model parameters. To circumvent this weakness, we present deep Bayesian neural networks for insurance pricing, a flexible machine learning framework that captures process and parameter uncertainties. By studying neural networks within the Bayesian framework, we can capture both sources of uncertainties, providing a better tool to diagnose when the model predictions are confident or not. We introduce the model, propose methods to estimate the parameters and present inference strategies. In addition, we establish a link between the proposed Bayesian framework and credibility theory: we, therefore, study the implications of using deep Bayesian neural networks within experience rating. Finally, we propose a method to interpret the predictions of deep Bayesian neural networks through prediction allocations based on Euler's principle.