Quality backlinks are the most important factors to determine if the website will ever rank or not. Although there are many different factors that influence your SEO, backlinks are the most important part. I would suggest you to work more on getting quality backlinks.
Since you are having a website in adult niche, I would suggest you to find links from adult sites. Related niche sites will be wonderful choice while getting your desired backlinks. Make sure you optimize your website properly for a wonderful user experience. Google analyzes how users interact with your website. So if you are site is not user-friendly and people don't spend much time on your site, getting backlinks won't do any good.
However if you can focus on getting more quality backlinks and wonderful user experience, you should be able to do pretty good in SEO.