Marriage solemnized according to Hindu rites subsequently registered under the Special Marriage Act, would not confer upon a spouse the right to annual marriage under Sec. 25(I) of the Special Marriage Act. Under Sec 25(I) willful refusal of a respondent spouse to consummate the marriage is a ground for annulling the marriage at the instance of the petitioner.
You can take divorce under such ground under Special Marriage Act, 1954 to Declare the Marriage Null and Void. On such ground a Hindu Marriage under the Hindu Marriage Act could be annulled although impotence at the time of marriage and its being continued till the institution of the proceeding was such a ground.
Disclaimer : This answer is based on information given by you hence, It is just opinion of Lawyer. For specific answer you have to give details and facts and choose the Lawyer of your choice.