Understanding the effects of mutations on protein stability is crucial for protein engineering, variant interpretation, and protein biophysics. While experimental measurements are the gold-standard, they can be time consuming, expensive and technically challenging. The data generated from these approaches, however, has driven an explosion in computational approaches to predict the effects of mutations on protein stability. Despite these efforts, community assessments of these tools have highlighted a range of limitations, including availability, high time costs, low predictive power, and biased predictions towards destabilizing mutations. To fill this gap, we developed DDMut, a fast and accurate siamese network to predict changes in Gibbs Free Energy (ΔΔG) upon single and multiple point mutations, leveraging both forward and the hypothetical reverse mutations to account for model anti-symmetry. The deep learning models were built by integrating graph-based representations of the localised 3D environment, with convolutional layers and transformer encoders. This combination could better capture the distance patterns between atoms by extracting both short-range and long-range interactions. Preliminary results show DDMut achieved pearson's correlations of up to 0.70 (RMSE: 1.37 kcal/mol) on predicting single point mutation by cross-validations, and 0.64 (RMSE: 1.80 kcal/mol) on multiple mutations, and outcompeted most available methods on non-redundant blind test sets. Importantly, DDMut was highly scalable and demonstrated anti-symmetric performance on both destabilizing and stabilizing mutations. We believe DDMut will be a useful platform to better understand the functional consequences of mutations, and guide rational protein engineering. DDMut is freely available as a web server and API at https://biosig.lab.uq.edu.au/ddmut.