In a large bowl, combine the condensed milk, vanilla extract, gingerbread spice and chocolate chips. Add a large dollop of whipped cream and mix until well combined. Add the remaining whipped cream and gently fold to combine. Add the crushed gingerbread men and fold to combine.