Volunteer corn in soybean fields are harmful as they disrupt the benefits of corn-soybean rotation. Volunteer corn does not only reduce soybean yield by competing for water, nutrition and sunlight, but also interferes with pest control (e.g., corn rootworm). It is therefore critical to monitor the volunteer corn in soybean at the crop seedling stage for better management. The current visual monitoring method is subjective and inefficient. Technology progress in sensing and automation provides a potential solution towards the automatic detection of volunteer corn from soybean. In this study, corn and soybean were planted in pots in greenhouse to mimic field conditions. Color images were collected by using a low-cost Intel RealSense camera for five successive days after the germination. Individual crops from images were manually cropped and subjected to image segmentation based on color threshold coupled with noise removal to create a dataset. Shape (i.e., area, aspect ratio, rectangularity, circularity, and eccentricity), color (i.e., R, G, B, H, S, V, L, a, b, Y, Cb, and Cr) and texture (coarseness, contrast, linelikeness, and directionality) features of individual crops were extracted. Individual feature's weights were ranked with the top 12 relevant features selected for this study. The 12 features were fed into three feature-based machine learning algorithms: support vector machine (SVM), neural network (NN) and random forest (RF) for model training. Prediction precision values on the test dataset for SVM, NN and RF were 85.3%, 81.6%, and 82.0%, respectively. The dataset (without feature extraction) was fed into two deep learning algorithms—GoogLeNet and VGG-16, resulting into 96.0% and 96.2% accuracies, respectively. The more satisfactory models from feature-based machine learning and deep learning were compared. VGG-16 was recommended for the purpose of distinguishing volunteer corn from soybean due to its higher detection accuracy, as well as smaller standard deviation (STD). This research demonstrated RGB images, coupled with VGG-16 algorithm could be used as a novel, reliable (accuracy >96%), and simple tool to detect volunteer corn from soybean. The research outcome helps provide critical information for farmers, agronomists, and plant scientists in monitoring volunteer corn infestation conditions in soybean for better decision making and management.