The reason Ye Qing didn't let the Sword Saint Sect Master come in actually stems from the cave suppressed by the Immovable Bodhisattva Seal.
Previously, outside the chamber where Patriarch Bodhidharma passed away, Ye Qing had seen such a cave and knew how dangerous it was. Inside the chamber where Patriarch Bodhidharma passed away, an Eternal Demon was imprisoned. The situation there is almost identical to this place. Moreover, when Ye Qing moved the Immovable Bodhisattva Seal slightly at that time, the chains dragged a very powerful dragon crocodile into the cave.
Ye Qing wasn't sure how formidable the demon inside the cave here was, but given it's suppressed by the Immovable Bodhisattva Seal, it certainly isn't simple. Ye Qing didn't let the Sword Saint Sect Master follow him for fear that the demon inside the cave might act again. If those chains activated once more and dragged the Sword Saint Sect Master in, it would be troublesome.
