But how could a complete sentence like 警戒心が剥き出し take the particle の to modify another noun?
Remember that Japanese has something called a "double subject" construction such as 彼は背が高い, フランスはワインが有名だ and この本は表紙が緑色だ. By pulling out the は-marked argument and leaving the が-marked argument, you can construct a relativized phrase like so:
- 背が高い人
tall people
- ワインが有名なフランス
France, which is famous for wine
- 表紙が緑色の本
a book with a green cover
Your sentence is basically an example of this. 警戒心が剥き出しの is a perfectly valid adjectival (noun-modifying) expression, and it modifies the following noun phrase 刺々しい雰囲気. (Note that むき出し is a no-adjective on its own, and Japanese adjectives are similar in syntax to verbs.)
To explain this in more detail, 彼女は警戒心が剥き出しだ is a valid sentence but (この)雰囲気は警戒心が剥き出しだ is not. I think this 警戒心が剥き出しの is a gapless relative clause that modifies (刺々しい)雰囲気. Similar examples include:
- カエルが池に飛び込む音
the sound of a frog jumping into a pond
- 誰かが魚を焼く煙
smoke from someone grilling a fish
- 彼が学校に行った帰り
on his way back from school