This paper describes how to construct a class specific hyper-graph (CSHG) model from a large corpus of multi-view images using local invariant features and their spatial configuration.As the first step of this method

each image is represented with a graph

which is constructed from a group of selected robust SIFT features.Secondly

a similarity propagation based graph clustering (SPGC) method is then proposed.Using this clustering method

the positive example graphs of a specific class accompanied with a set of negative example graphs are clustered into one or more clusters

which minimize an entropy function with a restriction defined on the F-measure.Based on SPGC and the rules of minimizing an entropy function

each cluster is simplified into a tree structure composed of a series of irreducible graphs.Finally

a recognition oriented class specific hyper-graph is generated from the given graph set.Using a trained CSHG model

object recognition can be implemented.Experimental results demonstrate the scalability and recognition performance of the method.