Neural network research in machine learning grew out of theories from computational neuroscience from the 1960s. While the class of affine-sigmoidal feature extractors has been studied extensively since the mid 1980s, the computational neuroscience community has moved on to new models that are qualitatively different and more descriptive, without being substantially more computationally expensive. This paper brings a particular model proposed in (Rust, 2005) for low level neurons in the macaque visual system into a machine-learning context : we evaluate their model as an activation function (feature-extractor) for single-layer neural networks that perform image classification. The function we evaluate is somewhat similar to the higher-order processing units discussed in (Minsky, 1969) and the Sigma-Pi units, but avoids the computational difficulties associated with these models by representing the second-order interaction weights with a low-rank positive semi-definite matrix, and avoids the learning difficulties associated with these models by using a gentler non-linearity than the logistic sigmoid. Remarkably good comparative results are obtained on three image classification tasks including 1.4% error on MNIST using a single-layer network. These results suggest that a single hidden layer neural network equipped with this neuron model can capture important patterns that escape standard models such as sigmoid neural networks and support vector machines based on gaussian and polynomial kernels.