How do I remove the dots / noise without damaging the text?

Question

I'm processing the images with OpenCV and Python. I need to remove the dots / noise from the image.
I tried dilation which made the dots smaller, however the text is being damaged. I also tried looping dilation twice and erosion once. But this did not give satisfactory results.
Is there some other way I can achieve this?
Thank you :)

EDIT:
I'm new to image processing. My current code is as follows

image = cv2.imread(file)
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
kernel = np.ones((2, 2), np.uint8)
gray = cv2.GaussianBlur(gray, (5, 5), 0)
gray = cv2.GaussianBlur(gray, (5, 5), 0)
gray = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 11, 2)
gray = cv2.erode(gray, kernel, iterations=1)
gray = cv2.dilate(gray, kernel, iterations=1)
cv2.imwrite(file.split('.'[0]+"_process.TIF", gray))

EDIT 2:
I tried median blurring. It has solved 90% of the issue. I had been using gaussianBlurring all this while.
Thank you

Answer 1

How about removing small connected components using connectedComponentsWithStats

import cv2
import numpy as np

img = cv2.imread('path_to_your_image', 0)
_, blackAndWhite = cv2.threshold(img, 127, 255, cv2.THRESH_BINARY_INV)

nlabels, labels, stats, centroids = cv2.connectedComponentsWithStats(blackAndWhite, None, None, None, 8, cv2.CV_32S)
sizes = stats[1:, -1] #get CC_STAT_AREA component
img2 = np.zeros((labels.shape), np.uint8)

for i in range(0, nlabels - 1):
    if sizes[i] >= 50:   #filter small dotted regions
        img2[labels == i + 1] = 255

res = cv2.bitwise_not(img2)

cv2.imwrite('res.png', res)

And here is c++ example:

Mat invBinarized;

threshold(inputImage, invBinarized, 127, 255, THRESH_BINARY_INV);
Mat labels, stats, centroids;

auto nlabels = connectedComponentsWithStats(invBinarized, labels, stats, centroids, 8, CV_32S, CCL_WU);

Mat imageWithoutDots(inputImage.rows, inputImage.cols, CV_8UC1, Scalar(0));
for (int i = 1; i < nlabels; i++) {
    if (stats.at<int>(i, 4) >= 50) {
        for (int j = 0; j < imageWithoutDots.total(); j++) {
            if (labels.at<int>(j) == i) {
                imageWithoutDots.data[j] = 255;
            }
        }
    }
}
cv::bitwise_not(imageWithoutDots, imageWithoutDots);

EDIT:
See also

OpenCV documentation for connectedComponentsWithStats

How to use openCV's connected components with stats in python

Example from learning opencv3