The invention relates to an infrared and visible-light different-source image matching method based on the context of line segments, comprising the following steps: using an LSD algorithm to detect line segments in an image, and selecting key line segments according to geometric constraint rules; detecting corners through an improved image corner detection method; calculating the Manhattan distance between the line segments in four-quadrant neighborhoods of a feature point to get the contribution of each line segment to the feature point, and on the basis, building a feature descriptor based on the context of line segments through adoption of a circular array; and using a bidirectional matching strategy and an RANSAC algorithm to realize infrared and visible-light image matching. Through the method, more correct matching point pairs can be acquired. The method can adapt to exact matching of infrared and visible-light images with serious gray level difference, and is superior to mainstream different-source image matching algorithms in terms of robustness and time efficiency.