In this paper, we formally present a novel estimation method, referred to as the Stochastic Learning Weak Estimator (SLWE), which is used to estimate the parameters of a binomial distribution, where the convergence of the estimate is weak, i.e. in law. The estimation is based on the principles of stochastic learning. Even though our new method includes a learning coefficient, λ, it turns out that the mean of the final estimate is independent of λ, the variance of the final distribution decreases with λ, and the speed decreases with λ. Similar results are true for the multinomial case. An empirical analysis on synthetic data shows the advantages of the scheme for non-stationary distributions. Conclusive results demonstrate the advantage of SLWE for a pattern-recognition problem which has direct implications in data compression. In this case, the underlying distribution in the data file to be compressed is non-stationary, and it is estimated and learnt using the principles highlighted here. By classifying its variation and using it in the compression, the superiority of the scheme is documented.