As the image data produced by individuals and enterprises is rapidly increasing, Scalar Invariant Feature Transform (SIFT), as a local feature detection algorithm, has been heavily employed in various areas, including object recognition, robotic mapping, etc. In this context, there is a growing need to outsource such image computation with high complexity to cloud for its economic computing resources and on-demand ubiquitous access. However, how to protect the private image data while enabling image computation becomes a major concern. To address this fundamental challenge, we study the privacy requirements in outsourcing SIFT computation and propose SecSIFT, a high performance privacy-preserving SIFT feature detection system. In previous private image computation works, one common approach is to encrypt the private image in a public key based homomorphic scheme that enables the original processing algorithms designed for plaintext domain to be performed over ciphertext domain. In contrast to these works, our system is not restricted by the efficiency limitations of homomorphic encryption scheme. The proposed system distributes the computation procedures of SIFT to a set of independent, co-operative cloud servers, and keeps the outsourced computation procedures as simple as possible to avoid utilizing homomorphic encryption scheme. Thus, it enables implementation with practical computation and communication complexity. Extensive experimental results demonstrate that SecSIFT performs comparably to original SIFT on image benchmarks while capable of preserving the privacy in an efficient way.